From pje at telecommunity.com Fri Dec 1 18:45:22 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 01 Dec 2006 12:45:22 -0500 Subject: [Distutils] Specifying a compiler In-Reply-To: References: <5.1.1.6.0.20061129232222.028b8cb0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061201124447.028bab48@sparrow.telecommunity.com> At 10:30 AM 12/1/2006 -0700, Robert McFadzean wrote: >Thank you for your help. I did download the MinGW compiler. When I try a >build with it I get: > >running build_ext >building '_mysql' extension >creating build\temp.win32-2.5 >creating build\temp.win32-2.5\Release >gcc -mno-cygwin -mdll -O -Wall -ID:/mysql/include -ID:\Python\include >-ID:\Pytho >n\PC -c _mysql.c -o build\temp.win32-2.5\Release\_mysql.o >/Dversion_info=(1,2,2, >'beta',2) /D__version__=1.2.2b2 >error: command 'gcc' failed: No such file or directory > >The gcc.exe file does exist in the MinGW\bin folder. Any ideas? You need to have that bin directory on your PATH when running setup scripts. From nathan at yergler.net Sat Dec 2 04:26:18 2006 From: nathan at yergler.net (Nathan R. Yergler) Date: Fri, 01 Dec 2006 22:26:18 -0500 Subject: [Distutils] develop egg recipe? Message-ID: <1165029978.5622.54.camel@Jane> I'm using zc.buildout for a project that also utilizes a custom egg-info writer. I specify the egg-info writer package as a setup_requires package in setup.py. Of course, that doesn't trigger zc.buildout (er, setuptools) to install that egg when I run buildout. I could work around this by specifying two parts in buildout.cfg: first install the egg for my egg-info writer, then create a development egg for my project (which uses the egg-info writer). I'm curious if there's a way with the current recipe set to create a develop egg within a part, as opposed to just specifying "develop = ." in the [buildout] section. Thanks, Nathan From jim at zope.com Sat Dec 2 14:50:36 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 2 Dec 2006 08:50:36 -0500 Subject: [Distutils] develop egg recipe? In-Reply-To: <1165029978.5622.54.camel@Jane> References: <1165029978.5622.54.camel@Jane> Message-ID: On Dec 1, 2006, at 10:26 PM, Nathan R. Yergler wrote: > I'm using zc.buildout for a project that also utilizes a custom egg- > info > writer. What is a custom egg-info writer? and why do you want one? > I specify the egg-info writer package as a setup_requires > package in setup.py. setuptools provided so many options like this. I really like setuptools, but worry that it's complexity will slow it's acceptance and slow the benefit from the essential simplicity of eggs. > Of course, that doesn't trigger zc.buildout (er, > setuptools) to install that egg when I run buildout. I wonder why. zc.buildout is just running the develop setup command. Is the setup_requires feature not expected to work with the develop command? > I could work > around this by specifying two parts in buildout.cfg: first install the > egg for my egg-info writer, then create a development egg for my > project > (which uses the egg-info writer). > > I'm curious if there's a way with the current recipe set to create a > develop egg within a part, as opposed to just specifying "develop = ." > in the [buildout] section. No. I plan to write a develop recipe, and am hoping to get to that this weekend. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Dec 2 17:29:08 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 02 Dec 2006 11:29:08 -0500 Subject: [Distutils] develop egg recipe? In-Reply-To: References: <1165029978.5622.54.camel@Jane> <1165029978.5622.54.camel@Jane> Message-ID: <5.1.1.6.0.20061202112455.028ce960@sparrow.telecommunity.com> At 08:50 AM 12/2/2006 -0500, Jim Fulton wrote: >On Dec 1, 2006, at 10:26 PM, Nathan R. Yergler wrote: > > > I'm using zc.buildout for a project that also utilizes a custom egg- > > info > > writer. > >What is a custom egg-info writer? and why do you want one? An egg-info writer allows you to generate files in .egg-info when the egg_info command is run. See: http://peak.telecommunity.com/DevCenter/setuptools#adding-new-egg-info-files > > Of course, that doesn't trigger zc.buildout (er, > > setuptools) to install that egg when I run buildout. > >I wonder why. zc.buildout is just running the develop >setup command. Is the setup_requires feature not expected >to work with the develop command? No, it's expected to work with develop. However, 'setup_requires' eggs are installed to the *setup directory*, not the installation target. This is always the case for setup_requires eggs, actually. So, if Nathan's "develop" is running (and invoking his egg_info writer), then everything is as expected. However, if the egg_info writer isn't running during "develop", then something is indeed wrong. From jim at zope.com Sun Dec 3 22:03:45 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 03 Dec 2006 16:03:45 -0500 Subject: [Distutils] buildout and zc.recipe.egg:custom problems In-Reply-To: <4564A5AC.1000706@zope.com> References: <45648F73.7010207@infrae.com> <4564A5AC.1000706@zope.com> Message-ID: <45733BB1.4090400@zope.com> Jim Fulton wrote: > Martijn Faassen wrote: >> Hi there, >> >> I'm trying to induce zc.buildout's zc.recipe.egg:custom recipe to build >> a library (lxml) against C libraries (libxml2, libxslt) installed in a >> custom place, i.e. as parts, such as parts/libxml2/include and >> parts/libxml2/lib. >> >> I thought I had that working before and even blogged about it, but it >> turns out that something's not entirely right yet. >> >> Right now I'm struggling to actually see an 'include-dirs' directive to >> have any effect on what's passed along to the C-compiler (namely the -I) >> option. It doesn't. After some research I think I've figured out why. >> I've installed lxml as a development egg, and somehow it always gets >> built the default way. The custom egg's install code is reached, but >> only when it's too late and lxml is already built. Therefore any special >> options such as include-dirs and library-dirs are never passed, as lxml >> has already been created. >> >> I figure lxml gets created automatically and not through the custom >> recipe as I have it listed as a develop egg in the [buildout] section. >> >> What to do about this? > > Unfortunately, you've hit a limitation in buildout and the custom recipe. > This limitation will, eventually, be removed. When creating develop eggs, > you have no control over build options. When creating custom eggs, > you have no way to use directories as source, and besides, you really want > a develop egg. > > I think what we want is a develop recipe that lets you specify various > options when creating develop eggs. I'll look at writing this over > the next few days. I still haven't finished this, but while working on it I realized that I should have suggested a work around. You can create a setup.cfg file in the directory containing your library setup.py file. This ConfigParser-style configuration file should have a build_ext section containing the options you want: [build_ext] include-dirs = /foo/include This is actually what the develop recipe will do behind the scenes. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From rdiaz02 at gmail.com Mon Dec 4 12:48:38 2006 From: rdiaz02 at gmail.com (Ramon Diaz-Uriarte) Date: Mon, 4 Dec 2006 12:48:38 +0100 Subject: [Distutils] checking if external dependencies are satisfied (and aborting ow). Message-ID: <624934630612040348q25851363x3824783faae748f@mail.gmail.com> (Apologies for cross-posting, but since this relates to both Pylons and distutils I figured it'd be OK) Dear All, We are developing a web-based application using Pylons. Pylons itself allows for easy distribution via eggs. Our application depends on non-Python software (mainly R and a bunch of R packages). As part of the installation, we would like to run a shell script (with possibly some Python and R code in it), and verify if all R-dependencies are satisfied, and if they are not, abort the installation. Note we are not trying to get setuptools to install the R-stuff, just run a script which checks a few things. I've searched the distutils and Pylons lists and docs and I cannot find this addressed, but I think I once read about this being possible. What am I missing? Thanks, R. -- Ramon Diaz-Uriarte Statistical Computing Team Structural Biology and Biocomputing Programme Spanish National Cancer Centre (CNIO) http://ligarto.org/rdiaz From jrfsousa at teor.fis.uc.pt Mon Dec 4 15:43:52 2006 From: jrfsousa at teor.fis.uc.pt (=?ISO-8859-1?Q?Jos=E9?= Rui Faustino de Sousa) Date: Mon, 04 Dec 2006 14:43:52 +0000 Subject: [Distutils] Data files and --prefix question. Message-ID: <1165243432.28665.6.camel@hopper.lca.uc.pt> Hi! I am writing a package installation using distutils it is very nice but I have it two snags: setup(... data_files=[("foo",["*.data"])],\ ...) setup.py install --prefix= a) Shouldn't the *.data syntax work for data_files as it does for package_data? b) Why does it install the data files on /foo and not in /share/foo like the documentation seems to imply? Everything else seems to work as advertised scripts go to /bin packages to /lib/python2.4/site-packages... Is it me or is it a bug? Thank you very much. Best regards, Jos? Rui From jim at zope.com Mon Dec 4 22:44:48 2006 From: jim at zope.com (Jim Fulton) Date: Mon, 4 Dec 2006 16:44:48 -0500 Subject: [Distutils] develop egg recipe? In-Reply-To: References: <1165029978.5622.54.camel@Jane> Message-ID: <48BF8B76-44AD-44DE-BE7E-A5F80D254BF7@zope.com> On Dec 2, 2006, at 8:50 AM, Jim Fulton wrote: ... > I plan to write a develop recipe, and am hoping to get to that this > weekend. I've just released a new version of zc.recipe.egg that provides a develop recipe: http://www.python.org/ pypi?:action=display&name=zc.recipe.egg&version=1.0.0b3#controlling- develop-egg-generation Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Dec 5 22:58:09 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 05 Dec 2006 16:58:09 -0500 Subject: [Distutils] checking if external dependencies are satisfied (and aborting ow). In-Reply-To: <624934630612040348q25851363x3824783faae748f@mail.gmail.com > Message-ID: <5.1.1.6.0.20061205165454.02aee0d0@sparrow.telecommunity.com> At 12:48 PM 12/4/2006 +0100, Ramon Diaz-Uriarte wrote: >(Apologies for cross-posting, but since this relates to both Pylons >and distutils I figured it'd be OK) > > >Dear All, > >We are developing a web-based application using Pylons. Pylons itself >allows for easy distribution via eggs. Our application depends on >non-Python software (mainly R and a bunch of R packages). As part of >the installation, we would like to run a shell script (with possibly >some Python and R code in it), and verify if all R-dependencies are >satisfied, and if they are not, abort the installation. > >Note we are not trying to get setuptools to install the R-stuff, just >run a script which checks a few things. I've searched the distutils >and Pylons lists and docs and I cannot find this addressed, but I >think I once read about this being possible. What am I missing? You may be thinking of the zc.buildout system, which allows you to create "recipes" (sort of like makefiles, but simpler) that can be used to install complex systems, including both eggs and non-Python software. See: http://cheeseshop.python.org/pypi/zc.buildout/ http://cheeseshop.python.org/pypi/zc.recipe.cmmi/ From pje at telecommunity.com Tue Dec 5 23:00:52 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 05 Dec 2006 17:00:52 -0500 Subject: [Distutils] Data files and --prefix question. In-Reply-To: <1165243432.28665.6.camel@hopper.lca.uc.pt> Message-ID: <5.1.1.6.0.20061205165816.02aedbd8@sparrow.telecommunity.com> At 02:43 PM 12/4/2006 +0000, Jos? Rui Faustino de Sousa wrote: >Hi! I am writing a package installation using distutils it is very nice >but I have it two snags: setup(... data_files=[("foo",["*.data"])],\ >...) setup.py install --prefix= a) Shouldn't the *.data syntax >work for data_files as it does for package_data? Perhaps. But data_files was a distutils option, and package_data was actually created in setuptools, which generally has options that are more "do-what-I-mean"ish than options that are native to the distutils. :) > b) Why does it install the data files on /foo and not in > /share/foo like the documentation seems to imply? Everything else > seems to work as advertised scripts go to /bin packages to > /lib/python2.4/site-packages... Is it me or is it a bug? I don't know. Personally I prefer to avoid data_files and use package_data instead, since that allows the package to access data at runtime, the data can be safely shipped in eggs, etc. From seefeld at sympatico.ca Wed Dec 6 05:28:36 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Tue, 05 Dec 2006 23:28:36 -0500 Subject: [Distutils] bdist_rpm fails due to bytecompiled modules In-Reply-To: <20061025213759.GA2273@night.wxnet.org> References: <20061025180500.GA27367@night.wxnet.org> <0D8C569F-5A77-4909-A153-24A083E47C18@mac.com> <20061025191319.GA30745@night.wxnet.org> <20061025213759.GA2273@night.wxnet.org> Message-ID: <457646F4.9090707@sympatico.ca> Christopher Blunck wrote: > distutils/setuptools creates .pyc files - that's ok. > > here's how the problem occurs: > > 1.) distutils generates a .spec file with an %install that calls > setup.py like below. .py and .pyc files are generated, and are > included in the INSTALLED_FILES record. see below for what is > called in %install: > > python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES > > > 2.) rpmbuild is called, and it generates a shell script that calls the > command above. AFTER that command is executed, rpmbuild calls some > strip commands as well as brp-python-bytecompile. > brp-python-bytecompile generates .pyo files from .pyc but it does not > update INSTALLED_FILES (because that is a distutils artifact that > rpmbuild doesn't know about) > > 3.) rpmbuild calls into %install, and INSTALLED_FILES is loaded. it > does not include the .pyo files that rpmbuild created. since > there are unpackaged files (the .pyo files) that exist in the > installation directory that are not referenced in the > INSTALLED_FILES file, rpmbuild chokes with an error. > > > step #3 is where the error occurs. it's basically an inconsistency > between distutils and rpmbuild. i can run distutils to generate > optimized modules (.pyo) but that doesn't apply to my scripts. as a > result i still have unpackaged files... Thanks for the detailed analysis ! It seems a workaround is to provide an install command that creates those .pyo files upfront, so they find their way into the INSTALLED_FILES list. Thus, I tried 'python setup.py bdist_rpm --install-script=install', where the content of 'install' is 'python setup.py install --optimize=1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES', and voila ! It works ! Not sure whether it is worth holding your breath for someone to fix distutils. :-( Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin... From pje at telecommunity.com Wed Dec 6 06:13:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 06 Dec 2006 00:13:39 -0500 Subject: [Distutils] bdist_rpm fails due to bytecompiled modules In-Reply-To: <457646F4.9090707@sympatico.ca> References: <20061025213759.GA2273@night.wxnet.org> <20061025180500.GA27367@night.wxnet.org> <0D8C569F-5A77-4909-A153-24A083E47C18@mac.com> <20061025191319.GA30745@night.wxnet.org> <20061025213759.GA2273@night.wxnet.org> Message-ID: <5.1.1.6.0.20061206001154.01f84e98@sparrow.telecommunity.com> At 11:28 PM 12/5/2006 -0500, Stefan Seefeld wrote: >Thus, I tried 'python setup.py bdist_rpm --install-script=install', >where the content of 'install' is >'python setup.py install --optimize=1 --root=$RPM_BUILD_ROOT >--record=INSTALLED_FILES', >and voila ! It works ! A simpler fix is to put: [install] optimize=1 in the project's setup.cfg, or perhaps better yet, in ~/.pydistutils.cfg or in $prefix/lib/python2.x/distutils/distutils.cfg. This will then always set --optimize=1 when install is invoked. From seefeld at sympatico.ca Wed Dec 6 06:24:30 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Wed, 06 Dec 2006 00:24:30 -0500 Subject: [Distutils] bdist_rpm fails due to bytecompiled modules In-Reply-To: <5.1.1.6.0.20061206001154.01f84e98@sparrow.telecommunity.com> References: <20061025213759.GA2273@night.wxnet.org> <20061025180500.GA27367@night.wxnet.org> <0D8C569F-5A77-4909-A153-24A083E47C18@mac.com> <20061025191319.GA30745@night.wxnet.org> <20061025213759.GA2273@night.wxnet.org> <5.1.1.6.0.20061206001154.01f84e98@sparrow.telecommunity.com> Message-ID: <4576540E.4040000@sympatico.ca> Phillip J. Eby wrote: > At 11:28 PM 12/5/2006 -0500, Stefan Seefeld wrote: >> Thus, I tried 'python setup.py bdist_rpm --install-script=install', >> where the content of 'install' is >> 'python setup.py install --optimize=1 --root=$RPM_BUILD_ROOT >> --record=INSTALLED_FILES', >> and voila ! It works ! > > A simpler fix is to put: > > [install] > optimize=1 > > in the project's setup.cfg, or perhaps better yet, in ~/.pydistutils.cfg > or in $prefix/lib/python2.x/distutils/distutils.cfg. This will then > always set --optimize=1 when install is invoked. I was thinking about that, but couldn't figure out whether the project's setup.cfg would be used when 'python setup.py install' is invoked from within rpmbuild. The docs are silent about this (and other) precious detail(s). Thanks for the tip ! Stefan -- ...ich hab' noch einen Koffer in Berlin... From rdiaz02 at gmail.com Wed Dec 6 16:05:57 2006 From: rdiaz02 at gmail.com (Ramon Diaz-Uriarte) Date: Wed, 6 Dec 2006 16:05:57 +0100 Subject: [Distutils] checking if external dependencies are satisfied (and aborting ow). In-Reply-To: <5.1.1.6.0.20061205165454.02aee0d0@sparrow.telecommunity.com> References: <5.1.1.6.0.20061205165454.02aee0d0@sparrow.telecommunity.com> Message-ID: <624934630612060705w76c64af0k9f23d817c4108b41@mail.gmail.com> On 12/5/06, Phillip J. Eby wrote: > At 12:48 PM 12/4/2006 +0100, Ramon Diaz-Uriarte wrote: > >(Apologies for cross-posting, but since this relates to both Pylons > >and distutils I figured it'd be OK) > > > > > >Dear All, > > > >We are developing a web-based application using Pylons. Pylons itself > >allows for easy distribution via eggs. Our application depends on > >non-Python software (mainly R and a bunch of R packages). As part of > >the installation, we would like to run a shell script (with possibly > >some Python and R code in it), and verify if all R-dependencies are > >satisfied, and if they are not, abort the installation. > > > >Note we are not trying to get setuptools to install the R-stuff, just > >run a script which checks a few things. I've searched the distutils > >and Pylons lists and docs and I cannot find this addressed, but I > >think I once read about this being possible. What am I missing? > > You may be thinking of the zc.buildout system, which allows you to create > "recipes" (sort of like makefiles, but simpler) that can be used to install > complex systems, including both eggs and non-Python software. See: > > http://cheeseshop.python.org/pypi/zc.buildout/ > http://cheeseshop.python.org/pypi/zc.recipe.cmmi/ > Dear Philip, Thanks a lot for the links!! This looks really nice and comprehensive. Best, R. > -- Ramon Diaz-Uriarte Statistical Computing Team Structural Biology and Biocomputing Programme Spanish National Cancer Centre (CNIO) http://ligarto.org/rdiaz From ilias at lazaridis.com Thu Dec 7 14:41:10 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Thu, 07 Dec 2006 15:41:10 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info Message-ID: I have the following use-case: within an application, i like to load several plugins from an folder, without any user-interaction app/plugins/pluginone app/plugins/plugintwo the plugins (standard sources prepared for setuptools, coming from svn) are arranged like this: pluginone/one pluginone/setup.py #standard setup.py file Is there any documented way to load such a "raw egg" (egg sources without a generated egg-info) into the application, without the need to generate the egg-info? . -- http://lazaridis.com From pje at telecommunity.com Thu Dec 7 16:55:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Dec 2006 10:55:43 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: Message-ID: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> At 03:41 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >Is there any documented way to load such a "raw egg" (egg sources >without a generated egg-info) into the application, without the need to >generate the egg-info? No. Of course, if you know where the plugin directories are, you can always run their "setup.py egg_info" to generate it. From Robert.McFadzean at lakelandc.ab.ca Wed Dec 6 19:59:30 2006 From: Robert.McFadzean at lakelandc.ab.ca (Robert McFadzean) Date: Wed, 6 Dec 2006 11:59:30 -0700 Subject: [Distutils] Compiler Incompatibility Message-ID: I have been trying to build MySQLdb for python and A. Dustman has come to the conclusion that the minGW32 compiler that I am using will not work with a Windows ( os is Windows XP) versions of Python and MySQL. Can anyone here confirm that he is right. Is there a possible work around? Dustman suggested that I would have to compile Python and MySQL with the gcc compiler for it to work. Robert G. McFadzean Instructor Vermilion School of Agriculture Lakeland College 5707 47 Ave West Vermilion AB T9X 1K5 Tlf: (780) 853-4890 Fax: (780) 853-8704 robert.mcfadzean at lakelandc.ab.ca From ilias at lazaridis.com Thu Dec 7 17:24:17 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Thu, 07 Dec 2006 18:24:17 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 03:41 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >> Is there any documented way to load such a "raw egg" (egg sources >> without a generated egg-info) into the application, without the need to >> generate the egg-info? > > No. Of course, if you know where the plugin directories are, you can > always run their "setup.py egg_info" to generate it. This is what I do, from within my programm, and it works fine. But I like to avoid this step. - I currently try to use the setuptools code to instantiate an egg from the setup.py file (which contains all information), and to apply then the functions from pkg_resources. Looks somehow like this: from distutils.core import run_setup dist = run_setup( setuppys[0],None,'init') print dist loaded_components = [] module = [] import pkg_resources for name in pkg_resources.get_entry_map(dist,'trac.plugins'): entry_point = pkg_resources.get_entry_info(dist,trac.plugins', name) entry_point.load() but this fails, I must have missed some point. (('Expected string, Requirement, or Distribution', )) although "dist" is an: - How can I create an egg representing object (from the sources/setup.py), from which I can load the entry-points afterwards, without having to generate an egg_info on the file-system? . -- http://lazaridis.com From pje at telecommunity.com Thu Dec 7 18:14:40 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Dec 2006 12:14:40 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> At 06:24 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >Looks somehow like this: > > >from distutils.core import run_setup >dist = run_setup( setuppys[0],None,'init') >print dist > >loaded_components = [] >module = [] > >import pkg_resources >for name in pkg_resources.get_entry_map(dist,'trac.plugins'): > entry_point = >pkg_resources.get_entry_info(dist,trac.plugins', name) > entry_point.load() > >but this fails, I must have missed some point. > >(('Expected string, Requirement, or Distribution', )) > >although "dist" is an: > > That Distribution is a distutils distribution, not a pkg_resources distribution. >How can I create an egg representing object (from the sources/setup.py), >from which I can load the entry-points afterwards, without having to >generate an egg_info on the file-system? You can take the dist.Distribution object's entry_points and parse it to create an entry point map. See http://peak.telecommunity.com/DevCenter/PkgResources#creating-and-parsing for details. In particular, you want to use: ep_map = EntryPoint.parse_map(dist.entry_points) entry_point = ep_map['trac.plugins'][name] There are several big drawbacks to this approach, because you're basically bypassing the entire eggs system. Here are two of the things that will break when you do this: * Plugins will not be able to depend on other plugins * Plugins will not be able to depend on other Python packages or projects I can't guarantee that there aren't other things that will break, too. I don't recommend that you use this approach in a published or production system, just because you don't like generating the egg-info on disk. It's there for several reasons, not the least of which is that it allows you to ship plugins as single .egg files. See also: http://peak.telecommunity.com/DevCenter/PkgResources#locating-plugins which shows how to do automatic plugin discovery and dependency resolution. This is another example of something that won't work with the approach you're trying to use. From lxander.m at gmail.com Thu Dec 7 19:55:37 2006 From: lxander.m at gmail.com (Alexander Michael) Date: Thu, 7 Dec 2006 13:55:37 -0500 Subject: [Distutils] SetupTools: In Situ Upgrading Message-ID: <525f23e80612071055j71a00226n412c635371a40cfe@mail.gmail.com> Once a script has started, is it possible to forcibly ignore the version of a package listed in easy-install.pth and tell pkg_resources to require a newer version that is available on the sys.path? The script actually installs the newer version using easy_install, but now I want the script to use the newer available version (and forget about the older version). Thanks for any hints, Alex By the way, it appears to me that in the EasyInstall documentation () the option --upgrade is elsewhere referred to as --update, but maybe it is my misunderstanding. From ilias at lazaridis.com Thu Dec 7 19:57:19 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Thu, 07 Dec 2006 20:57:19 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 06:24 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >> Looks somehow like this: ... >> > > That Distribution is a distutils distribution, not a pkg_resources > distribution. ok >> How can I create an egg representing object (from the sources/setup.py), >>from which I can load the entry-points afterwards, without having to >> generate an egg_info on the file-system? > > You can take the dist.Distribution object's entry_points and parse it to > create an entry point map. > > See > http://peak.telecommunity.com/DevCenter/PkgResources#creating-and-parsing > for details. In particular, you want to use: > > ep_map = EntryPoint.parse_map(dist.entry_points) > entry_point = ep_map['trac.plugins'][name] Although not the prefered solution, I've tried this: import pkg_resources from distutils.core import run_setup dist = run_setup( setuppys[0],None,'init') ep_map = pkg_resources.EntryPoint.parse_map(dist.entry_points, dist) ep_list = ep_map['trac.plugins'] print ep_list for name, entry_point in ep_list.iteritems(): print name print entry_point try: entry_point.load() except pkg_resources.DistributionNotFound, e: ... File "C:\prg\py24\lib\site-packages\setuptools-0.6c3-py2.4.egg\pkg_resources.py", line 1829, in load if require: self.require(env, installer) File "C:\prg\py24\lib\site-packages\setuptools-0.6c3-py2.4.egg\pkg_resources.py", line 1842, in require working_set.resolve(self.dist.requires(self.extras),env,installer)) AttributeError: Distribution instance has no attribute 'requires' > There are several big drawbacks to this approach, because you're basically > bypassing the entire eggs system. ... (elaborations) . -- http://lazaridis.com From ilias at lazaridis.com Thu Dec 7 20:03:46 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Thu, 07 Dec 2006 21:03:46 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 06:24 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >> Looks somehow like this: ... >> How can I create an egg representing object (from the sources/setup.py), >>from which I can load the entry-points afterwards, without having to >> generate an egg_info on the file-system? > > You can take the dist.Distribution object's entry_points and parse it to > create an entry point map. ... > I can't guarantee that there aren't other things that will break, too. I ... (elaborations) I understand, and I basicly would like to use the setuptools egg code. This is a special and limited use-case: The eggs do *not* need to be found by other eggs/applications, as they are used only by the application that loads them in. Thus I just want to create the egg representation in my code (to call the entry_points), whilst using the information from setup.py/sources (instead of the generated egg_info). Essentially, the code (internal in setuptools) that reads the egg_info will simply retrieve it from the setup.py (or i instantiate it somhow, and pass it to the setuptools routines) The function would be essentially: def get_an_egg(path_to_setuppy) # load setup.py # construct the egg return egg #further processing based on standard egg methods . -- http://lazaridis.com From pje at telecommunity.com Thu Dec 7 20:37:58 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Dec 2006 14:37:58 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061207143613.01f8cf10@sparrow.telecommunity.com> At 08:57 PM 12/7/2006 +0200, Ilias Lazaridis wrote: > File >"C:\prg\py24\lib\site-packages\setuptools-0.6c3-py2.4.egg\pkg_resources.py", >line 1829, in load > if require: self.require(env, installer) Yeah, you need to pass 'require=False' to entrypoint.load(). See: http://peak.telecommunity.com/DevCenter/PkgResources#entrypoint-objects As I said, dependency resolution for these non-egg things isn't going to work right. From ilias at lazaridis.com Thu Dec 7 20:45:14 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Thu, 07 Dec 2006 21:45:14 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> Message-ID: Ilias Lazaridis wrote: > Phillip J. Eby wrote: >> At 06:24 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >>> Looks somehow like this: > ... > >>> How can I create an egg representing object (from the sources/setup.py), >> >from which I can load the entry-points afterwards, without having to >>> generate an egg_info on the file-system? >> You can take the dist.Distribution object's entry_points and parse it to >> create an entry point map. > ... >> I can't guarantee that there aren't other things that will break, too. I > ... (elaborations) > > I understand, and I basicly would like to use the setuptools egg code. > > This is a special and limited use-case: > > The eggs do *not* need to be found by other eggs/applications, as they > are used only by the application that loads them in. > > Thus I just want to create the egg representation in my code (to call > the entry_points), whilst using the information from setup.py/sources > (instead of the generated egg_info). > > Essentially, the code (internal in setuptools) that reads the egg_info > will simply retrieve it from the setup.py (or i instantiate it somhow, > and pass it to the setuptools routines) > > The function would be essentially: > > def get_an_egg(path_to_setuppy) > # load setup.py > # construct the egg > return egg > > #further processing based on standard egg methods - I am a little overworked at this point, but this should be the right direction to enable setuptools to create distributions whilst using the information within setup.py (and without breaking any behaviour): in pkg_resources def register_finder(importer_type, distribution_finder): """Register `distribution_finder` to find distributions in sys.path items _distribution_finders[importer_type] = distribution_finder def find_distributions(path_item, only=False): """Yield distributions accessible via `path_item`""" importer = get_importer(path_item) finder = _find_adapter(_distribution_finders, importer) return finder(importer, path_item, only) . -- http://lazaridis.com From pje at telecommunity.com Thu Dec 7 20:47:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Dec 2006 14:47:12 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> At 09:03 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >This is a special and limited use-case: > >The eggs do *not* need to be found by other eggs/applications, as they >are used only by the application that loads them in. That's 100% irrelevant. There are a whole bunch of different ways for you to install them so that they will be used only by your application. "easy_install -md /path/to/plugins" is one of the simplest, but you can also drop .egg files directly there, or run a plugin's "setup.py develop -md /path/to/plugins" (which allows it to run from the source), etc. Once you have plugins installed (or deposited) in a plugins directory, have your app use find_plugins(), add them to sys.path, and then just load the entry points: http://peak.telecommunity.com/DevCenter/PkgResources#locating-plugins Note that in both cases, the -m option (--multi-version) is important for installing or developing plugins, as that allows you to install to a non-sys.path directory and doesn't require .pth support. >Thus I just want to create the egg representation in my code (to call >the entry_points), whilst using the information from setup.py/sources >(instead of the generated egg_info). I still don't get why you can't run "develop", "egg_info", or build .egg files. If the answer is simply that you don't want to, I'd suggest getting over it. :) If there's some other reason, you haven't shared it with me yet. I suspect, however, that the actual problem is that you're not aware of one or more of the several other ways to implement plugins... that all actually work. :) From ilias at lazaridis.com Thu Dec 7 20:48:15 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Thu, 07 Dec 2006 21:48:15 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061207143613.01f8cf10@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143613.01f8cf10@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 08:57 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >> File >> "C:\prg\py24\lib\site-packages\setuptools-0.6c3-py2.4.egg\pkg_resources.py", >> line 1829, in load >> if require: self.require(env, installer) > > Yeah, you need to pass 'require=False' to entrypoint.load(). See: > > http://peak.telecommunity.com/DevCenter/PkgResources#entrypoint-objects > > As I said, dependency resolution for these non-egg things isn't going to > work right. Setuptools seems to be a very huge work. I appreciate your efforts in having made my live (as a user) with python easier. Thank's a lot for your assistance! (will sleep a little, hopefully tomorrow things become more clear.) . -- http://lazaridis.com From pje at telecommunity.com Thu Dec 7 20:53:30 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Dec 2006 14:53:30 -0500 Subject: [Distutils] SetupTools: In Situ Upgrading In-Reply-To: <525f23e80612071055j71a00226n412c635371a40cfe@mail.gmail.co m> Message-ID: <5.1.1.6.0.20061207144721.0284e420@sparrow.telecommunity.com> At 01:55 PM 12/7/2006 -0500, Alexander Michael wrote: >Once a script has started, is it possible to forcibly ignore the >version of a package listed in easy-install.pth and tell pkg_resources >to require a newer version that is available on the sys.path? Yes, but it's only possible with a crude hack at the moment. Explicitly add your new version to the beginning of sys.path, then call pkg_resources.working_set.__init__(). This will clear and reset the global working set to match the current state of sys.path. From pje at telecommunity.com Thu Dec 7 21:09:31 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Dec 2006 15:09:31 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061207150121.0296f440@sparrow.telecommunity.com> At 09:45 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >I am a little overworked at this point, but this should be the right >direction to enable setuptools to create distributions whilst using the >information within setup.py (and without breaking any behaviour): The only way I'll ever support this is if it's basically a way to automatically run the egg_info command on demand (perhaps by checking whether setup.py is newer than the .egg-info/PKG-INFO file). There are huge areas of functionality that will otherwise break -- and I'm not going to spend thirty minutes trying to write down all the ones I can think of, let alone the ones I can't think of right now. I've already given you the workarounds for you to do it in your own system, but remember that if anything breaks, it's broken. The minute that any of your plugins depend on *anything* else, or expect to use metadata generated by egg_info, the game is over. So it's never going to be a supported feature of setuptools unless the egg_info command gets run. From pje at telecommunity.com Thu Dec 7 21:15:58 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Dec 2006 15:15:58 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> At 08:57 PM 12/7/2006 +0200, Ilias Lazaridis wrote: > ep_map = pkg_resources.EntryPoint.parse_map(dist.entry_points, >dist) By the way, I missed this line in your email before: you should *not* pass dist as the second argument here. That method is looking for a pkg_resources.Distribution, not a distutils.Distribution. distutils.Distribution (and setuptools.dist.Distribution) represent *source* distributions that have not been built yet. pkg_resources.Distribution objects represent *built* and *importable* distributions. They have different data structures and behaviors. distutils Distribution objects have data and methods for running commands and building, while pkg_resources.Distribution objects know about import locations, entry points, package versions, and things like that. You can't pass a distutils Distribution into *any* pkg_resources APIs, or vice versa -- they are utterly incompatible objects. From sjmachin at lexicon.net Thu Dec 7 21:32:40 2006 From: sjmachin at lexicon.net (John Machin) Date: Fri, 08 Dec 2006 07:32:40 +1100 Subject: [Distutils] Compiler Incompatibility In-Reply-To: References: Message-ID: <45787A68.6070707@lexicon.net> On 7/12/2006 5:59 AM, Robert McFadzean wrote: > I have been trying to build MySQLdb for python and A. Dustman has come to the conclusion that the minGW32 compiler that I am using will not work with a Windows ( os is Windows XP) versions of Python and MySQL. Can anyone here confirm that he is right. Is there a possible work around? Dustman suggested that I would have to compile Python and MySQL with the gcc compiler for it to work. > I know nothing about mysqldb, but the above is confused/confusing: The mingw32 compiler is *a* gcc compiler. What is *the* gcc compiler to which you refer? Why build anything? Are there not Windows binaries already available? I'd suggest that you seek the advice of the mysqldb maintainer :-) From chris at kateandchris.net Thu Dec 7 23:14:17 2006 From: chris at kateandchris.net (Chris Lambacher) Date: Thu, 7 Dec 2006 17:14:17 -0500 Subject: [Distutils] Compiler Incompatibility In-Reply-To: References: Message-ID: <20061207221417.GC19400@kateandchris.net> On Wed, Dec 06, 2006 at 11:59:30AM -0700, Robert McFadzean wrote: > I have been trying to build MySQLdb for python and A. Dustman has come to the conclusion that the minGW32 compiler that I am using will not work with a Windows ( os is Windows XP) versions of Python and MySQL. Can anyone here confirm that he is right. Is there a possible work around? Dustman suggested that I would have to compile Python and MySQL with the gcc compiler for it to work. A quick search finds a plethora of resources for how to use mingw to compile python extentions using the standard Windows Python distribution. http://www.google.ca/search?q=python+compiler+mingw32&ie=utf-8&oe=utf-8&rls=com.ubuntu:en-US:official&client=firefox-a Many of those are out of date and some steps can be skipped and either the export file is actually shipped with 2.4 and 2.5 or mingw32-gcc just deals with it interally. I haven't done this in a while, but, I think all you really need to do at this point is: python setup.py build --compiler=mingw32 python setup.py install Note, you can make the '--compiler=mingw32' a default setting by adding """ [build] compiler=mingw32 """ to c:\Python2x\Lib\distutils\distutils.cfg, at which point all you have to do is: python setup.py install I don't know any details particular to building MySQLdb. -Chris > > Robert G. McFadzean > Instructor > Vermilion School of Agriculture > Lakeland College > 5707 47 Ave West > Vermilion AB T9X 1K5 > Tlf: (780) 853-4890 > Fax: (780) 853-8704 > robert.mcfadzean at lakelandc.ab.ca > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From ilias at lazaridis.com Fri Dec 8 12:42:53 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 13:42:53 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 08:57 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >> ep_map = pkg_resources.EntryPoint.parse_map(dist.entry_points, >> dist) > By the way, I missed this line in your email before: you should *not* pass > dist as the second argument here. That method is looking for a > pkg_resources.Distribution, not a distutils.Distribution. ok, this one works now (btw: either with "dist" or without it. but i've removed it, as you've suggested). > distutils.Distribution (and setuptools.dist.Distribution) represent > *source* distributions that have not been built yet. ok > pkg_resources.Distribution objects represent *built* and *importable* > distributions. They have different data structures and > behaviors. distutils Distribution objects have data and methods for > running commands and building, while pkg_resources.Distribution objects > know about import locations, entry points, package versions, and things > like that. You can't pass a distutils Distribution into *any* > pkg_resources APIs, or vice versa -- they are utterly incompatible objects. I have missed the existence of "pkg_resources.Distribution" totally (was looking on "setuptools.dist.Distribution / disutils.Distribution" ) are "pkg_resources.Distribution" objects finally the internal representation of eggs? . -- http://lazaridis.com From pje at telecommunity.com Fri Dec 8 14:09:34 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Dec 2006 08:09:34 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061208080918.0279f1e0@sparrow.telecommunity.com> At 01:42 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >are "pkg_resources.Distribution" objects finally the internal >representation of eggs? Yes. From ilias at lazaridis.com Fri Dec 8 14:35:00 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 15:35:00 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061208080918.0279f1e0@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> <5.1.1.6.0.20061208080918.0279f1e0@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 01:42 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >> are "pkg_resources.Distribution" objects finally the internal >> representation of eggs? > > Yes. ok. Possibly the class should be named "pkg_resources.Egg" or "pkg_resources.EggDistribution" in order to become more descriptive. - (sidenote: I am just finshing some code, thus I can point you to some examples of the usage-context) . -- http://lazaridis.com From ilias at lazaridis.com Fri Dec 8 16:55:25 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 17:55:25 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061207150121.0296f440@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207150121.0296f440@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 09:45 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >> I am a little overworked at this point, but this should be the right >> direction to enable setuptools to create distributions whilst using the >> information within setup.py (and without breaking any behaviour): > > The only way I'll ever support this ... > So it's never going to be a supported > feature of setuptools unless the egg_info command gets run. My question was from an architectural nature. If someone (for _any_ reason) want's to avoid generation of egg-info files, would he use the mechnism below to pass a finder/loader to setuptools which does not rely on the egg-info? def register_finder(importer_type, distribution_finder): def find_distributions(path_item, only=False): . -- http://lazaridis.com From ilias at lazaridis.com Fri Dec 8 17:24:00 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 18:24:00 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 09:03 PM 12/7/2006 +0200, Ilias Lazaridis wrote: >> This is a special and limited use-case: >> >> The eggs do *not* need to be found by other eggs/applications, as they >> are used only by the application that loads them in. > > That's 100% irrelevant. ok > There are a whole bunch of different ways for you > to install them [...] I understand. Flexibility is very nice. But simplicity, too. My requirement is to have no additional installation steps after an svn-checkout (or unzip). >> Thus I just want to create the egg representation in my code (to call >> the entry_points), whilst using the information from setup.py/sources >> (instead of the generated egg_info). > > I still don't get why you can't run "develop", "egg_info", or build .egg > files. If the answer is simply that you don't want to, I'd suggest getting > over it. :) If there's some other reason, you haven't shared it with me yet. The reason is "reduncancy", which I like to avoid. For simple plugins, the information is already given within "setup.py", thus duplicating it into an egg-info is "duplicating information". > I suspect, however, that the actual problem is that you're not aware of one > or more of the several other ways to implement plugins... that all actually > work. :) Trac uses setuptools as it's plugin mechanism, thus I have to deal with eggs in my use case. Most (if not all) of those plugins just use the basic constructs of setuptools. In my use-case, a loader retrieves all 2nd-level directories from within a folder and loads the eggs: http://dev.lazaridis.com/base/browser/infra/tracx/loader-draft-test.py?rev=171 I had initially placed the egg-info-generation within this code, but this leads to problems with permissions etc. - and: it's redundant. - As said, as a user I am very happy with setuptools. As a developer, I would like to have more simplicity for simple use cases. I think removing the need for an egg-info would be the first step. - How does this read: Setuptools allows you to use simple python packages as eggs, without the need to generate an egg-info. This is especially usefull in development environments or in environments with a high amount of plugins. The information needed will be retrieved from within the setup.py file. If a feature requires generation of the egg-info, you'll recieve either an warning, and error or the egg-info will be generated automaticly (depending on configuration). This way, any folder which contains an setup.py file containing the configuration can serve as an "raw egg". . -- http://lazaridis.com From jnelson-distutils at securepipe.com Fri Dec 8 18:11:59 2006 From: jnelson-distutils at securepipe.com (Jon Nelson) Date: Fri, 08 Dec 2006 11:11:59 -0600 Subject: [Distutils] setuptools and bdist_rpm Message-ID: <45799CDF.4070504@securepipe.com> I'm new to setuptools and I'm hoping to have a few questions answered. I also have some observations that may be useful, but that can come later. 1. why does setuptools bdist_rpm use --single-version-externally-managed? 2. is it possible to have setuptools write dependency information not just to requires.txt but also to setup.cfg so that RPM will be aware of the dependencies? 3. This is more of a distutils question, but, what can be done about the INSTALLED_FILES containing directories that aren't marked with %dir (this causes two problems: /all/ directories end up being created but NOT owned by the RPM, and all files /under/ that directory get included, which means they likely get included *twice* to to also being explicitly included). Sorry for the multi-issue email, I'm new at this. From ronaldoussoren at mac.com Fri Dec 8 18:29:59 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 8 Dec 2006 18:29:59 +0100 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> Message-ID: On Dec 8, 2006, at 5:24 PM, Ilias Lazaridis wrote: > > My requirement is to have no additional installation steps after an > svn-checkout (or unzip). You could include the egg-info in your repository and distribution archive. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20061208/d668b63f/attachment-0001.bin From pje at telecommunity.com Fri Dec 8 18:45:14 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Dec 2006 12:45:14 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061208080918.0279f1e0@sparrow.telecommunity.com> <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207151245.04005e78@sparrow.telecommunity.com> <5.1.1.6.0.20061208080918.0279f1e0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061208124410.027a1600@sparrow.telecommunity.com> At 03:35 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >Possibly the class should be named "pkg_resources.Egg" or >"pkg_resources.EggDistribution" in order to become more descriptive. No. It's also used to track source distributions being downloaded from PyPI, and even .win32.exe distributions. It's not limited to eggs alone, nor to any one format even of eggs. From pje at telecommunity.com Fri Dec 8 18:55:29 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Dec 2006 12:55:29 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> At 06:24 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >My requirement is to have no additional installation steps after an >svn-checkout (or unzip). Then generate the .egg-info before committing to SVN, or use the "sdist" command to create your source zipfiles or tarballs, and all will be well. >The reason is "reduncancy", which I like to avoid. > >For simple plugins, the information is already given within "setup.py", >thus duplicating it into an egg-info is "duplicating information". Perhaps you should then also patch Python to avoid generating .pyc and .pyo files, since these are redundant in the same way and for the same reason. :) >As said, as a user I am very happy with setuptools. As a developer, I >would like to have more simplicity for simple use cases. I think >removing the need for an egg-info would be the first step. Generated egg-info is a feature, not a bug. Note that in order to use the information from the setup.py, you must import huge amounts of setuptools and distutils code, whereas normal egg operations require only the single pkg_resources module. If a project uses custom egg-info or other setuptools extensions, you must import those as well. All of this, to avoid writing some data to disk that could then be read simply and directly without involving any of those other imports! The redundancy you're complaining about is a major enhancement to startup performance. As I've said, an ability to auto-generate egg_info if a setup.py is newer than the corresponding PKG-INFO might be a useful feature. However, trying to do without generating it at all, simply isn't going to happen. From pje at telecommunity.com Fri Dec 8 18:58:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Dec 2006 12:58:36 -0500 Subject: [Distutils] setuptools and bdist_rpm In-Reply-To: <45799CDF.4070504@securepipe.com> Message-ID: <5.1.1.6.0.20061208125549.03fdcd00@sparrow.telecommunity.com> At 11:11 AM 12/8/2006 -0600, Jon Nelson wrote: >I'm new to setuptools and I'm hoping to have a few questions answered. I >also have some observations that may be useful, but that can come later. > >1. why does setuptools bdist_rpm use > --single-version-externally-managed? To avoid the need for .pth files, mainly. >2. is it possible to have setuptools write dependency information not >just to requires.txt but also to setup.cfg so that RPM will be aware of >the dependencies? Not really, but you could create a custom egg_info writer that will reconfigure the bdist_rpm command at runtime based on that information. See the setuptools documentation for more about egg-info writers. You'll need some understanding of the distutils command framework to do this effectively. >3. This is more of a distutils question, but, what can be done about the >INSTALLED_FILES containing directories that aren't marked with %dir >(this causes two problems: /all/ directories end up being created but >NOT owned by the RPM, and all files /under/ that directory get included, >which means they likely get included *twice* to to also being explicitly >included). Sorry, no clue on that one. From ilias at lazaridis.com Fri Dec 8 19:24:00 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 20:24:00 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> References: <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 06:24 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >> My requirement is to have no additional installation steps after an >> svn-checkout (or unzip). > > Then generate the .egg-info before committing to SVN, or use the "sdist" > command to create your source zipfiles or tarballs, and all will be well. There is still redundancy (and an additional step). >> The reason is "reduncancy", which I like to avoid. >> >> For simple plugins, the information is already given within "setup.py", >> thus duplicating it into an egg-info is "duplicating information". > > Perhaps you should then also patch Python to avoid generating .pyc and .pyo > files, since these are redundant in the same way and for the same reason. :) I understand this analogy, but I think you exaggregate here. >> As said, as a user I am very happy with setuptools. As a developer, I >> would like to have more simplicity for simple use cases. I think >> removing the need for an egg-info would be the first step. > > Generated egg-info is a feature, not a bug. Note that in order to use the > information from the setup.py, you must import huge amounts of setuptools > and distutils code, whereas normal egg operations require only the single > pkg_resources module. The information in setup.py (at least for simple eggs/plugins) is simple data. Retrieving a few values directly from the setup.py (instead from an egg-info) would be fast and without any dependency. > If a project uses custom egg-info or other setuptools extensions, you must > import those as well. All of this, to avoid writing some data to disk that > could then be read simply and directly without involving any of those other > imports! The redundancy you're complaining about is a major enhancement to > startup performance. eggs can cover 'light' needs and 'heavy' needs. Possibly for the 'heavy' needs its an enhancement. But for the 'light' needs, the overall overhead is (from my point of view) inacceptable. I go even a step further: I think python is powerfull enouth to do all this 'natively' (e.g. module attributes etc.), an without a major speed loss, even in more complex cases. > As I've said, an ability to auto-generate egg_info if a setup.py is newer > than the corresponding PKG-INFO might be a useful feature. However, trying > to do without generating it at all, simply isn't going to happen. I understand. Thus the only way to avoid "egg-info" is to avoid setuptools. - I am collecting requirements on this page, possibly you are interested: http://case.lazaridis.com/wiki/Package . -- http://lazaridis.com From pje at telecommunity.com Fri Dec 8 19:55:25 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Dec 2006 13:55:25 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061208132736.03f57250@sparrow.telecommunity.com> At 08:24 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >Retrieving a few values directly from the setup.py (instead from an >egg-info) would be fast and without any dependency. In order to invoke run_setup to get the data, you must import, *at absolute minimum*: distutils distutils.cmd distutils.core distutils.dist distutils.util distutils.errors distutils.command distutils.command.install distutils.command.sdist distutils.command.install_lib setuptools setuptools.dist setuptools.depends setuptools.extension setuptools.command setuptools.command.install setuptools.command.sdist setuptools.command.install_lib Pyrex.Distutils.build_ext (if it's installed) And I didn't even go through all of these modules to see what *they* import. It also ignores any customization features usage. And you want to do all this, to avoid writing cache files? > > If a project uses custom egg-info or other setuptools extensions, you must > > import those as well. All of this, to avoid writing some data to disk > that > > could then be read simply and directly without involving any of those > other > > imports! The redundancy you're complaining about is a major > enhancement to > > startup performance. > >eggs can cover 'light' needs and 'heavy' needs. > >Possibly for the 'heavy' needs its an enhancement. > >But for the 'light' needs, the overall overhead is (from my point of >view) inacceptable. That's because you're not paying attention to anyone's needs but your own. The use cases you consider "heavy" may be considered "light" by other people. Anything that one person does not happen to need, they will consider "heavy". Someone who doesn't use entry points, for example, will consider *your* use of them to be "heavy", even as you consider their other egg_info uses to be "heavy" instead of "light", as they consider those uses to be. Thus, your view of the situation is extremely self-centered, especially since you've received plenty of help in how to minimize the overhead -- it requires you to run a single command before checkin or source distribution, and *only* in the case where you have changed your version number or entry points. If that's too much work for you, then this is indeed the wrong toolset for you. You should instead spend large amounts of time developing and testing your own system, so that you can save that handful of keystrokes needed to type "setup.py egg_info" every once in a while. Good luck to you on that. :) > > As I've said, an ability to auto-generate egg_info if a setup.py is newer > > than the corresponding PKG-INFO might be a useful feature. However, > trying > > to do without generating it at all, simply isn't going to happen. > >I understand. > >Thus the only way to avoid "egg-info" is to avoid setuptools. Correct. Your choices are as follows: 1. get used to typing "setup.py egg_info" when you change your project's version number or entry points 2. write a script or alias to do it for you whenever you check your code in, or 3. reinvent the wheel so you can save a minute or two doing one of the above. If you choose #3, I sure hope it is because you're doing this for a hobby or personal project, because if you aren't, then you are defrauding your employer or client by diverting their funds for your personal enjoyment at wheel reinventing, while providing them with no additional benefits. From ilias at lazaridis.com Fri Dec 8 20:22:57 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 21:22:57 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061208132736.03f57250@sparrow.telecommunity.com> References: <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> <5.1.1.6.0.20061208132736.03f57250@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 08:24 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >> Retrieving a few values directly from the setup.py (instead from an >> egg-info) would be fast and without any dependency. > > In order to invoke run_setup to get the data, you must import, *at absolute > minimum*: > > distutils ... I must read-in(!) at absolute minimum just the "setup.py" file, without any includes. ... >> But for the 'light' needs, the overall overhead is (from my point of >> view) inacceptable. > > That's because you're not paying attention to anyone's needs but your > own. ... > Thus, your view of the situation is extremely self-centered, ... of course I look at my requirements. (whilst taking in account that others exist. thats btw. why I not say "drop those egg-infos") >>> As I've said, an ability to auto-generate egg_info if a setup.py is newer >>> than the corresponding PKG-INFO might be a useful feature. However, >> trying >>> to do without generating it at all, simply isn't going to happen. >> I understand. >> >> Thus the only way to avoid "egg-info" is to avoid setuptools. > > Correct. Your choices are as follows: > > 1. get used to typing "setup.py egg_info" when you change your project's > version number or entry points I've not choosen a dynamic language to do redundant steps. Otherwise I could pick java, which needs flagship-IDE's to stay managable. I like simplicity - that's why I am with python. > 2. write a script or alias to do it for you whenever you check your code in, or I do not provide automations, in order to manage redundant steps (only as a temporary solution) > 3. reinvent the wheel so you can save a minute or two doing one of the above. a minute or two, thousands of developers = many time. > If you choose #3, I sure hope it is because you're doing this for a hobby > or personal project, because if you aren't, then you are defrauding your > employer or client by diverting their funds for your personal enjoyment at > wheel reinventing, while providing them with no additional benefits. as said, from a user-level, I rate your work as _very_ helpfull. As a developer, I've mentione a deficit (from my point of view) and I just like to have some flexibility. I hope that a few other developers have a similar view and realize the need o "raw eggs" or "lightweight egg-based plugins", thus we can workout a solution. If anyone is interested, please contact me. This is a starting point, nothing else: http://dev.lazaridis.com/base/browser/infra/tracx/loader-draft-test.py?rev=171 The requirements as a whole: http://case.lazaridis.com/wiki/Package . -- http://lazaridis.com From pje at telecommunity.com Fri Dec 8 20:57:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Dec 2006 14:57:39 -0500 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: <5.1.1.6.0.20061208132736.03f57250@sparrow.telecommunity.com> <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> <5.1.1.6.0.20061208132736.03f57250@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061208145556.042c8e60@sparrow.telecommunity.com> At 09:22 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >Phillip J. Eby wrote: > > At 08:24 PM 12/8/2006 +0200, Ilias Lazaridis wrote: > >> Retrieving a few values directly from the setup.py (instead from an > >> egg-info) would be fast and without any dependency. > > > > In order to invoke run_setup to get the data, you must import, *at > absolute > > minimum*: > > > > distutils >... > >I must read-in(!) at absolute minimum just the "setup.py" file, without >any includes. Then you are confused about how run_setup() works. run_setup() invokes the setup.py, which causes all of those modules to be imported. Perhaps you intend to instead parse the setup.py and figure out what it does from its syntax tree, without actually executing it? Good luck on that as well. From ilias at lazaridis.com Fri Dec 8 21:16:43 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 22:16:43 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: <5.1.1.6.0.20061208145556.042c8e60@sparrow.telecommunity.com> References: <5.1.1.6.0.20061208132736.03f57250@sparrow.telecommunity.com> <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207105442.02826f00@sparrow.telecommunity.com> <5.1.1.6.0.20061207120454.027a0ff8@sparrow.telecommunity.com> <5.1.1.6.0.20061207143804.04bde208@sparrow.telecommunity.com> <5.1.1.6.0.20061208124701.049c87a0@sparrow.telecommunity.com> <5.1.1.6.0.20061208132736.03f57250@sparrow.telecommunity.com> <5.1.1.6.0.20061208145556.042c8e60@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 09:22 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >> Phillip J. Eby wrote: >>> At 08:24 PM 12/8/2006 +0200, Ilias Lazaridis wrote: >>>> Retrieving a few values directly from the setup.py (instead from an >>>> egg-info) would be fast and without any dependency. >>> In order to invoke run_setup to get the data, you must import, *at >> absolute >>> minimum*: >>> >>> distutils >> ... >> >> I must read-in(!) at absolute minimum just the "setup.py" file, without >> any includes. > > Then you are confused about how run_setup() works. run_setup() invokes the > setup.py, which causes all of those modules to be imported. I'm not refering to "run_setup". > Perhaps you intend to instead parse the setup.py and figure out what it > does from its syntax tree, without actually executing it? Good luck on > that as well. You are speculating much to complex. Possibly during the long work on setuptools you've forgotten a few basic things: * python is very dynamic and expressive. * python is simple. * python stays simple, except you make things complex. - I have to sleep now. A good night (day) to all! . -- http://case.lazaridis.com/wiki/Package From ilias at lazaridis.com Fri Dec 8 21:17:19 2006 From: ilias at lazaridis.com (Ilias Lazaridis) Date: Fri, 08 Dec 2006 22:17:19 +0200 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: Message-ID: Ilias Lazaridis wrote: > I have the following use-case: > > within an application, i like to load several plugins from an folder, > without any user-interaction > > app/plugins/pluginone > app/plugins/plugintwo > > the plugins (standard sources prepared for setuptools, coming from svn) > are arranged like this: > > pluginone/one > pluginone/setup.py #standard setup.py file > > Is there any documented way to load such a "raw egg" (egg sources > without a generated egg-info) into the application, without the need to > generate the egg-info? As a summary to this thread: I hope that a few other developers have a similar view and realize the need o "raw eggs" or "lightweight egg-based plugins", thus we can workout a solution. If anyone is interested, please contact me. This is a starting point, nothing else: http://dev.lazaridis.com/base/browser/infra/tracx/loader-draft-test.py?rev=171 The requirements as a whole: http://case.lazaridis.com/wiki/Package . -- http://lazaridis.com From chris at kateandchris.net Fri Dec 8 23:17:46 2006 From: chris at kateandchris.net (Chris Lambacher) Date: Fri, 8 Dec 2006 17:17:46 -0500 Subject: [Distutils] Compiler Incompatibility In-Reply-To: References: <20061207221417.GC19400@kateandchris.net> Message-ID: <20061208221746.GA28449@kateandchris.net> This is not a problem with compiling python extensions, this is a problem linking to the mysqlclient library. Check out the comments on this page: http://dev.mysql.com/doc/refman/5.1/en/windows-client-compiling.html Unfortunately I can't try anything out for you since I don't have a Windows machine anymore (changed jobs). -Chris On Fri, Dec 08, 2006 at 02:24:58PM -0700, Robert McFadzean wrote: > I have tried a number of the suggestions on getting the minGW32 compiler to build python extensions, but I still get the same hangup which seems to be here: > > d:/mysql/lib/opt\mysqlclient.lib(./release/my_thr_init.obj)(.text[_my_thread_glo > bal_init]+0x1e):C:\cygwin\home\mys: variable '_iob' can't be auto-imported. Plea > se read the documentation for ld's --enable-auto-import for details. > > And I don't know what that means other than that it proceeds with a lot of warnings, tries something else, and then gives it up. > > Robert G. McFadzean > Instructor > Vermilion School of Agriculture > Lakeland College > 5707 47 Ave West > Vermilion AB T9X 1K5 > Tlf: (780) 853-4890 > Fax: (780) 853-8704 > robert.mcfadzean at lakelandc.ab.ca > > > -----Original Message----- > From: Chris Lambacher [mailto:chris at kateandchris.net] > Sent: Thursday, December 07, 2006 3:14 PM > To: Robert McFadzean > Cc: distutils-sig at python.org > Subject: Re: [Distutils] Compiler Incompatibility > > > On Wed, Dec 06, 2006 at 11:59:30AM -0700, Robert McFadzean wrote: > > I have been trying to build MySQLdb for python and A. Dustman has come to the conclusion that the minGW32 compiler that I am using will not work with a Windows ( os is Windows XP) versions of Python and MySQL. Can anyone here confirm that he is right. Is there a possible work around? Dustman suggested that I would have to compile Python and MySQL with the gcc compiler for it to work. > A quick search finds a plethora of resources for how to use mingw to compile > python extentions using the standard Windows Python distribution. > http://www.google.ca/search?q=python+compiler+mingw32&ie=utf-8&oe=utf-8&rls=com.ubuntu:en-US:official&client=firefox-a > > Many of those are out of date and some steps can be skipped and either the > export file is actually shipped with 2.4 and 2.5 or mingw32-gcc just deals > with it interally. I haven't done this in a while, but, I think all you > really need to do at this point is: > > python setup.py build --compiler=mingw32 > python setup.py install > > Note, you can make the '--compiler=mingw32' a default setting by adding > """ > [build] > compiler=mingw32 > """ > to c:\Python2x\Lib\distutils\distutils.cfg, at which point all you have to do > is: > python setup.py install > > I don't know any details particular to building MySQLdb. > > -Chris > > > > > Robert G. McFadzean > > Instructor > > Vermilion School of Agriculture > > Lakeland College > > 5707 47 Ave West > > Vermilion AB T9X 1K5 > > Tlf: (780) 853-4890 > > Fax: (780) 853-8704 > > robert.mcfadzean at lakelandc.ab.ca > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig From elvelind at gmail.com Sun Dec 10 20:28:00 2006 From: elvelind at gmail.com (Elvelind Grandin) Date: Sun, 10 Dec 2006 20:28:00 +0100 Subject: [Distutils] setuptools extras_require Message-ID: <5035593a0612101128x42542df3ia26e348489222141@mail.gmail.com> Is there any way to exclude packages in extras_require as opposed to just include them. In Turbogears we have Sqlobject in by default. but if the extra "future" is used it should drop SO and install sqlalchemy instead. -- cheers elvelind grandin From pje at telecommunity.com Sun Dec 10 21:58:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 10 Dec 2006 15:58:07 -0500 Subject: [Distutils] setuptools extras_require In-Reply-To: <5035593a0612101128x42542df3ia26e348489222141@mail.gmail.co m> Message-ID: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: >Is there any way to exclude packages in extras_require as opposed to >just include them. In Turbogears we have Sqlobject in by default. but >if the extra "future" is used it should drop SO and install sqlalchemy >instead. Not currently, no. Do you have any suggestions as to how this might be spelled in the requirements API? From lxander.m at gmail.com Mon Dec 11 15:43:50 2006 From: lxander.m at gmail.com (Alexander Michael) Date: Mon, 11 Dec 2006 09:43:50 -0500 Subject: [Distutils] SetupTools: What Files Were Installed? Message-ID: <525f23e80612110643g78b2532eu775d4c9f9a9be8a4@mail.gmail.com> Currently, I am easy_install'ing with --record=FILENAME set to save the list of installed filenames in a known location so that I can later visit it to determine what needs to be removed in order to uninstall the package. It would be nice to reduce the footprint on the filesystem by not leaving this trace, but is it possible or even a good idea? Is there a re-usable bit of the SetupTools internals that would allow me to recreate the contents of this file dynamically on-demand without reinstalling? If not, then in the simple packages that I have been working with, there are two things that need to be found a deleted: 1) The egg file or directory in site-packages. 2) Any console or GUI scripts in the Scripts/bin directory. I can find the egg file or directory by looking at pkg_resources.Distribution.location, but how to find the scripts? What about a non-default script location setting? Again, maybe it is not even a good idea to attempt to avoid --record=FILENAME. Your thoughts are appreciated. Thanks, Alex From pje at telecommunity.com Mon Dec 11 17:12:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Dec 2006 11:12:47 -0500 Subject: [Distutils] SetupTools: What Files Were Installed? In-Reply-To: <525f23e80612110643g78b2532eu775d4c9f9a9be8a4@mail.gmail.co m> Message-ID: <5.1.1.6.0.20061211110936.027f1988@sparrow.telecommunity.com> At 09:43 AM 12/11/2006 -0500, Alexander Michael wrote: >Currently, I am easy_install'ing with --record=FILENAME set to save >the list of installed filenames in a known location so that I can >later visit it to determine what needs to be removed in order to >uninstall the package. It would be nice to reduce the footprint on >the filesystem by not leaving this trace, but is it possible or even a >good idea? > >Is there a re-usable bit of the SetupTools internals that would allow >me to recreate the contents of this file dynamically on-demand without >reinstalling? Sort of. There's no official API, but the data is more-or-less there. See below. >If not, then in the simple packages that I have been working with, >there are two things that need to be found a deleted: > >1) The egg file or directory in site-packages. >2) Any console or GUI scripts in the Scripts/bin directory. > >I can find the egg file or directory by looking at >pkg_resources.Distribution.location, but how to find the scripts? What >about a non-default script location setting? The scripts' names can be determined from the entry points and the 'scripts' metadata directory. Their link to the specific egg(s) can be confirmed by inspecting the comment lines at the beginning of the script, as easy_install always writes a comment describing the type of script and the egg it came from. So, in principle you can check script directories for scripts of the right name(s) and verify their contents to ensure they are actually associated with the egg you plan to remove, and not just another script with the same name. From lxander.m at gmail.com Mon Dec 11 17:34:50 2006 From: lxander.m at gmail.com (Alexander Michael) Date: Mon, 11 Dec 2006 11:34:50 -0500 Subject: [Distutils] SetupTools: What Files Were Installed? In-Reply-To: <5.1.1.6.0.20061211110936.027f1988@sparrow.telecommunity.com> References: <5.1.1.6.0.20061211110936.027f1988@sparrow.telecommunity.com> Message-ID: <525f23e80612110834q3b3f50e8ob56a5aafa78441cf@mail.gmail.com> On 12/11/06, Phillip J. Eby wrote: > At 09:43 AM 12/11/2006 -0500, Alexander Michael wrote: > >Currently, I am easy_install'ing with --record=FILENAME set to save > >the list of installed filenames in a known location so that I can > >later visit it to determine what needs to be removed in order to > >uninstall the package. It would be nice to reduce the footprint on > >the filesystem by not leaving this trace, but is it possible or even a > >good idea? > > > >Is there a re-usable bit of the SetupTools internals that would allow > >me to recreate the contents of this file dynamically on-demand without > >reinstalling? > > Sort of. There's no official API, but the data is more-or-less there. See > below. > > > >If not, then in the simple packages that I have been working with, > >there are two things that need to be found a deleted: > > > >1) The egg file or directory in site-packages. > >2) Any console or GUI scripts in the Scripts/bin directory. > > > >I can find the egg file or directory by looking at > >pkg_resources.Distribution.location, but how to find the scripts? What > >about a non-default script location setting? > > The scripts' names can be determined from the entry points and the > 'scripts' metadata directory. Their link to the specific egg(s) can be > confirmed by inspecting the comment lines at the beginning of the script, > as easy_install always writes a comment describing the type of script and > the egg it came from. > > So, in principle you can check script directories for scripts of the right > name(s) and verify their contents to ensure they are actually associated > with the egg you plan to remove, and not just another script with the same > name. I started out doing this, but I had two concerns: 1) Is it safe to assume a one-to-one mapping between entry points and script files and to recreate naming conventions (including entrypoint-script.py on Windows)? 2) How do I determine the --script-dir? It sounds like it is safe to assume 1, but that 2 is indeterminate (as easy_install does not leave it's own trace about the user overrides at the time of installation). The result being that there is no generic mechanism to find the files that need to be deleted in order to uninstall a package without resorting to requiring that the package was installed with --record=FILENAME. I appreciate all your help! Alex From pje at telecommunity.com Mon Dec 11 17:41:49 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Dec 2006 11:41:49 -0500 Subject: [Distutils] SetupTools: What Files Were Installed? In-Reply-To: <525f23e80612110834q3b3f50e8ob56a5aafa78441cf@mail.gmail.co m> References: <5.1.1.6.0.20061211110936.027f1988@sparrow.telecommunity.com> <5.1.1.6.0.20061211110936.027f1988@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061211113925.03e5f008@sparrow.telecommunity.com> At 11:34 AM 12/11/2006 -0500, Alexander Michael wrote: >It sounds like it is safe to assume 1, but that 2 is indeterminate (as >easy_install does not leave it's own trace about the user overrides at >the time of installation). It's indeterminate, but verifiable. That is, you can simply look at PATH or other such directories and tell whether a given script came from the egg. > The result being that there is no generic >mechanism to find the files that need to be deleted in order to >uninstall a package without resorting to requiring that the package >was installed with --record=FILENAME. Right; the "nest" command framework will have a mechanism to deal with this; nests will know what files they've installed, and nest will provide complete package management facilities for the files involved. From elvelind at gmail.com Mon Dec 11 22:44:53 2006 From: elvelind at gmail.com (Elvelind Grandin) Date: Mon, 11 Dec 2006 22:44:53 +0100 Subject: [Distutils] setuptools extras_require In-Reply-To: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> Message-ID: <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> Not really extras_require = { 'PDF': ["ReportLab>=1.2", "RXP"], } extras_exclude = { 'PDF': ["Foo"], } might work. even if it's not that pretty. On 12/10/06, Phillip J. Eby wrote: > At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: > >Is there any way to exclude packages in extras_require as opposed to > >just include them. In Turbogears we have Sqlobject in by default. but > >if the extra "future" is used it should drop SO and install sqlalchemy > >instead. > > Not currently, no. Do you have any suggestions as to how this might be > spelled in the requirements API? > > -- cheers elvelind grandin From bronger at physik.rwth-aachen.de Tue Dec 12 09:27:08 2006 From: bronger at physik.rwth-aachen.de (Torsten Bronger) Date: Tue, 12 Dec 2006 09:27:08 +0100 Subject: [Distutils] trouble with data_files + RPM Message-ID: <87hcw1bjtv.fsf@wilson.homeunix.com> Hall?chen! I try to distribute a Python application with i18n files (.mo). My setup.py says: ... data_files = [('share/locale/de/LC_MESSAGES', ['po/de/felo.mo'])], packages = ['felo'], package_dir = {'felo': 'src'}, The directory structure seen from setup.py is: ./setup.py po/ src/ This works perfectly with the bdist_dumb command. However, when I try to generate an RPM, it says: ... running install_data creating /var/tmp/Felo-1.0-1-buildroot/usr/share creating /var/tmp/Felo-1.0-1-buildroot/usr/share/locale creating /var/tmp/Felo-1.0-1-buildroot/usr/share/locale/de creating /var/tmp/Felo-1.0-1-buildroot/usr/share/locale/de/LC_MESSAGES error: can't copy 'po/de/felo.mo': doesn't exist or not a regular file error: Bad exit status from /var/tmp/rpm-tmp.6388 (%install) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.6388 (%install) error: command 'rpmbuild' failed with exit status 1 What can I do about it? Thank you! Tsch?, Torsten. -- Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646 (See http://ime.webhop.org for Jabber, MSN, etc.) From jean.jordaan at gmail.com Tue Dec 12 15:57:53 2006 From: jean.jordaan at gmail.com (Jean Jordaan) Date: Tue, 12 Dec 2006 16:57:53 +0200 Subject: [Distutils] workingenv: always installs setuptools? Message-ID: <91d4ce220612120657m327d8f25y818999260fa59228@mail.gmail.com> Hi there In install_setuptools I read: "(only called if setuptools is not installed on the system)". However, the only call to install_setuptools is unconditional, and it wants to install a local copy of setuptools although I have it installed. Which is wrong, the docstring or the code? This is with revision 2128. -- jean . .. .... //\\\oo///\\ From jorge.vargas at gmail.com Tue Dec 12 16:21:45 2006 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Tue, 12 Dec 2006 11:21:45 -0400 Subject: [Distutils] setuptools extras_require In-Reply-To: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> Message-ID: <32822fe60612120721ldc13c3kc406da0c0e9d4eec@mail.gmail.com> On 12/10/06, Phillip J. Eby wrote: > At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: > >Is there any way to exclude packages in extras_require as opposed to > >just include them. In Turbogears we have Sqlobject in by default. but > >if the extra "future" is used it should drop SO and install sqlalchemy > >instead. I'm not sure if extras is the correct way to do this, since the current TG setup is I want sqlobject OR sqlalchemy, extras is more like addons. maybe it will need a bigger structure to handle it. > > Not currently, no. Do you have any suggestions as to how this might be > spelled in the requirements API? I like the way gentoo handles it on portage. there is this thing call "use flags" which are just a label everyone agrees on it's meaning. so the tg package will have something like IUSE="sqlobject sqlalchemy" so in setup() under install_requires it will do a check like sqlobject? ("SQLObject >= 0.7.1,<=0.7.99") sqlalchemy? ("SQLAlchemy >= 0.3") then (and setuptools doesn't has this) on the pre_install() function you will have something like. if not sqlalchemy in params: set sqlobject as default. and of course sqlobject or sqlalchemy needs to be pass in to determine what you need. now portage is a bigger system since it's supposed to handle compilation of any languaje and USE flags where an abstraction to c/makefile variables to provide the --with,--with-out,etc. I'm not sure if this is a feature python/setuptools needs right now but in the future we'll see many more packages depending on optional backends. or interchangable components. just my 2cents > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From jorge.vargas at gmail.com Tue Dec 12 16:29:23 2006 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Tue, 12 Dec 2006 11:29:23 -0400 Subject: [Distutils] SETUPTOOLS - Loading Eggs which do not have an egg_info In-Reply-To: References: Message-ID: <32822fe60612120729t4585ededy4bf0da2b28e15284@mail.gmail.com> hi troll PS: just making sure other people in this list know about it PSS: please do not reply On 12/8/06, Ilias Lazaridis wrote: > Ilias Lazaridis wrote: From elvelind at gmail.com Tue Dec 12 16:32:17 2006 From: elvelind at gmail.com (Elvelind Grandin) Date: Tue, 12 Dec 2006 16:32:17 +0100 Subject: [Distutils] setuptools extras_require In-Reply-To: <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> Message-ID: <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> Or use a magic extra name like "default" which is used if no other extra is selected. On 12/11/06, Elvelind Grandin wrote: > Not really > > extras_require = { > 'PDF': ["ReportLab>=1.2", "RXP"], > } > extras_exclude = { > 'PDF': ["Foo"], > } > > might work. even if it's not that pretty. > > On 12/10/06, Phillip J. Eby wrote: > > At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: > > >Is there any way to exclude packages in extras_require as opposed to > > >just include them. In Turbogears we have Sqlobject in by default. but > > >if the extra "future" is used it should drop SO and install sqlalchemy > > >instead. > > > > Not currently, no. Do you have any suggestions as to how this might be > > spelled in the requirements API? > > > > > > > -- > cheers > elvelind grandin > -- cheers elvelind grandin From jorge.vargas at gmail.com Tue Dec 12 16:55:23 2006 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Tue, 12 Dec 2006 11:55:23 -0400 Subject: [Distutils] setuptools extras_require In-Reply-To: <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> Message-ID: <32822fe60612120755l718c3479rcd9466458fc82643@mail.gmail.com> On 12/12/06, Elvelind Grandin wrote: > Or use a magic extra name like "default" which is used if no other > extra is selected. umm if we make that a list then it could work, it could even be used to get template engines. > > On 12/11/06, Elvelind Grandin wrote: > > Not really > > > > extras_require = { > > 'PDF': ["ReportLab>=1.2", "RXP"], > > } > > extras_exclude = { > > 'PDF': ["Foo"], > > } > > > > might work. even if it's not that pretty. > > > > On 12/10/06, Phillip J. Eby wrote: > > > At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: > > > >Is there any way to exclude packages in extras_require as opposed to > > > >just include them. In Turbogears we have Sqlobject in by default. but > > > >if the extra "future" is used it should drop SO and install sqlalchemy > > > >instead. > > > > > > Not currently, no. Do you have any suggestions as to how this might be > > > spelled in the requirements API? > > > > > > > > > > > > -- > > cheers > > elvelind grandin > > > > > -- > cheers > elvelind grandin > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From bhendrix at enthought.com Tue Dec 12 20:18:17 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Tue, 12 Dec 2006 13:18:17 -0600 Subject: [Distutils] warnings while merging eggs with namespace packages Message-ID: <457F0079.1050207@enthought.com> Please forgive me if this has been asked before, I thought it had been, but I can't find it on the list now... We've been trying to switch our python distribution over to eggs, but when importing multiple eggs that share the same base namespace, we get tons of warnings "Module %s was already imported from %, but %s is being added to sys.path." I can see where this might be a valid message, so I don't want to filter it out always, but bombarding the user with 40 duplicate warnings when they just start up the interpreter is not acceptable either. Ideally, we'd get this message one time per namespace. Are there any suggestions as how we could implement this behavior? Is this grounds for a feature request? Thanks, Bryce From dangoor at gmail.com Tue Dec 12 22:47:41 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 12 Dec 2006 16:47:41 -0500 Subject: [Distutils] setuptools extras_require In-Reply-To: <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> Message-ID: <454ABBD2-4737-4EFE-B21E-5560BA05AF43@gmail.com> On Dec 12, 2006, at 10:32 AM, Elvelind Grandin wrote: > Or use a magic extra name like "default" which is used if no other > extra is selected. +1 on default as a list. TurboGears has its own setup program (based on ez_setup.py) for installing/upgrading setuptools and then installing the "right" TurboGears (where right could mean a version in testing with different dependencies and whatnot). By having a default set of extras, then "easy_install TurboGears" can still work for those that are savvy to easy_install. Kevin From matt at matt-good.net Tue Dec 12 17:35:01 2006 From: matt at matt-good.net (Matt Good) Date: Tue, 12 Dec 2006 11:35:01 -0500 Subject: [Distutils] setuptools extras_require In-Reply-To: <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> Message-ID: <1165941302.6958.23.camel@nny> On Tue, 2006-12-12 at 16:32 +0100, Elvelind Grandin wrote: > Or use a magic extra name like "default" which is used if no other > extra is selected. This doesn't really address the full issue, since there may be several sets of alternative dependencies for different requirements. Following the TurboGears example it needs an ORM from "SQLObject or SQLAlchemy" and a template engine from "Kid or Genshi". So it'd need defaults for each, not just a single default. > On 12/11/06, Elvelind Grandin wrote: > > Not really > > > > extras_require = { > > 'PDF': ["ReportLab>=1.2", "RXP"], > > } > > extras_exclude = { > > 'PDF': ["Foo"], > > } > > > > might work. even if it's not that pretty. > > > > On 12/10/06, Phillip J. Eby wrote: > > > At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: > > > >Is there any way to exclude packages in extras_require as opposed to > > > >just include them. In Turbogears we have Sqlobject in by default. but > > > >if the extra "future" is used it should drop SO and install sqlalchemy > > > >instead. > > > > > > Not currently, no. Do you have any suggestions as to how this might be > > > spelled in the requirements API? > > > > > > > > > > > > -- > > cheers > > elvelind grandin > > > > From matt at matt-good.net Thu Dec 14 08:52:05 2006 From: matt at matt-good.net (Matt Good) Date: Thu, 14 Dec 2006 02:52:05 -0500 Subject: [Distutils] setuptools extras_require In-Reply-To: <5035593a0612132215u3c08dda0h7f2f6fd9011c1697@mail.gmail.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> <1165941302.6958.23.camel@nny> <5035593a0612132215u3c08dda0h7f2f6fd9011c1697@mail.gmail.com> Message-ID: <1166082725.27648.50.camel@nny> On Thu, 2006-12-14 at 07:15 +0100, Elvelind Grandin wrote: > Perhaps I wasn't totally clear. what I meant was something like this. > > extras_require = { > "default" : ["SQLObject==bugfix,>=0.7.1dev-r1860,<=0.7.99"], > "future" : ["Genshi>=0.3", "SQLAlchemy>=0.3"] > }, That may be ok for the example, but what happens if you add another extra "foo" for some optional feature. Installing "TurboGears[foo]" won't select either the "default" or "future" extras, so you don't really get all the requirements. >From a semantic perspective I think listing these as "extras" is odd. One of either SQLObject or SQLAlchemy is required for installation, so it's not just an "extra" feature. I think extras have been used for this as a workaround to setuptools not supporting ORed requirements, but I'd like to see that fixed. I tend to think in terms of Debian packages where stuff like SA or SO would be listed as "depends" meaning that they're strictly required, so in setuptools this would correspond to "install_requires". On the other hand I believe Paste is still an optional package used in TurboGears for stuff like the interactive traceback display, so it'd be a Debian "recommends", or setuptools "extras". Using "|" as a separator for ORing TurboGear's requirements might look something like this (ignoring version numbers for simplicity): install_requires = [ "SQLObject | SQLAlchemy", "Kid | Genshi", ], extras_require = { "fancy_tb": ["Paste"], } Setuptools would check if either SQLObject or SQLAlchemy was already installed, if not it'd try from left to right until it successfully installed one of the requirements. I suppose extras could still be used as aliases for something like the "future" requirements to make it easier for users to install this set of packages: extras_require = { "standard": ["Kid", "SQLObject"], "future": ["Genshi", "SQLAlchemy"], "fancy_tb": ["Paste"], } Alternatively tuples could be substituted to define ORed requirements like: install_requires = [ ("SQLObject", "SQLAlchemy"), ("Kid", "Genshi"), ] Comments? -- Matt Good > On 12/12/06, Matt Good wrote: > > On Tue, 2006-12-12 at 16:32 +0100, Elvelind Grandin wrote: > > > Or use a magic extra name like "default" which is used if no other > > > extra is selected. > > > > This doesn't really address the full issue, since there may be several > > sets of alternative dependencies for different requirements. > > > > Following the TurboGears example it needs an ORM from "SQLObject or > > SQLAlchemy" and a template engine from "Kid or Genshi". So it'd need > > defaults for each, not just a single default. > > > > > On 12/11/06, Elvelind Grandin wrote: > > > > Not really > > > > > > > > extras_require = { > > > > 'PDF': ["ReportLab>=1.2", "RXP"], > > > > } > > > > extras_exclude = { > > > > 'PDF': ["Foo"], > > > > } > > > > > > > > might work. even if it's not that pretty. > > > > > > > > On 12/10/06, Phillip J. Eby wrote: > > > > > At 08:28 PM 12/10/2006 +0100, Elvelind Grandin wrote: > > > > > >Is there any way to exclude packages in extras_require as opposed to > > > > > >just include them. In Turbogears we have Sqlobject in by default. but > > > > > >if the extra "future" is used it should drop SO and install sqlalchemy > > > > > >instead. > > > > > > > > > > Not currently, no. Do you have any suggestions as to how this might be > > > > > spelled in the requirements API? > > > > > > > > > > > > > > > > > > > > > > -- > > > > cheers > > > > elvelind grandin > > > > > > > > > > > > > > > > From fairwinds at eastlink.ca Thu Dec 14 22:49:24 2006 From: fairwinds at eastlink.ca (David Pratt) Date: Thu, 14 Dec 2006 17:49:24 -0400 Subject: [Distutils] Compilation and eggification of swig code In-Reply-To: <1166082725.27648.50.camel@nny> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> <1165941302.6958.23.camel@nny> <5035593a0612132215u3c08dda0h7f2f6fd9011c1697@mail.gmail.com> <1166082725.27648.50.camel@nny> Message-ID: <4581C6E4.2070605@eastlink.ca> Hi. I'd like to produce an egg by compiling swig code for a particular platform. Can someone advise a blueprint for this. Many thanks. Regards, David From pje at telecommunity.com Fri Dec 15 01:01:56 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 14 Dec 2006 19:01:56 -0500 Subject: [Distutils] Compilation and eggification of swig code In-Reply-To: <4581C6E4.2070605@eastlink.ca> References: <1166082725.27648.50.camel@nny> <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> <1165941302.6958.23.camel@nny> <5035593a0612132215u3c08dda0h7f2f6fd9011c1697@mail.gmail.com> <1166082725.27648.50.camel@nny> Message-ID: <5.1.1.6.0.20061214185921.029e89e0@sparrow.telecommunity.com> At 05:49 PM 12/14/2006 -0400, David Pratt wrote: >Hi. I'd like to produce an egg by compiling swig code for a particular >platform. Can someone advise a blueprint for this. Many thanks. See http://python.org/doc/2.4.1/dist/dist.html and http://python.org/doc/2.4.1/dist/describing-extensions.html especially section 2.3.2, which notes that you can use .i files as Extension() sources. Add the necessary information to your setup.py and run bdist_egg on the appropriate platform(s) to create the egg(s) you want. From fairwinds at eastlink.ca Fri Dec 15 18:20:18 2006 From: fairwinds at eastlink.ca (David Pratt) Date: Fri, 15 Dec 2006 13:20:18 -0400 Subject: [Distutils] Compilation and eggification of swig code In-Reply-To: <5.1.1.6.0.20061214185921.029e89e0@sparrow.telecommunity.com> References: <1166082725.27648.50.camel@nny> <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> <1165941302.6958.23.camel@nny> <5035593a0612132215u3c08dda0h7f2f6fd9011c1697@mail.gmail.com> <1166082725.27648.50.camel@nny> <5.1.1.6.0.20061214185921.029e89e0@sparrow.telecommunity.com> Message-ID: <4582D952.9080500@eastlink.ca> Many thanks for this Phillip. I'll give this a try. Regards, David Phillip J. Eby wrote: > At 05:49 PM 12/14/2006 -0400, David Pratt wrote: >> Hi. I'd like to produce an egg by compiling swig code for a particular >> platform. Can someone advise a blueprint for this. Many thanks. > > See http://python.org/doc/2.4.1/dist/dist.html and > http://python.org/doc/2.4.1/dist/describing-extensions.html > > especially section 2.3.2, which notes that you can use .i files as > Extension() sources. Add the necessary information to your setup.py and > run bdist_egg on the appropriate platform(s) to create the egg(s) you want. > From lobais at gmail.com Sat Dec 16 17:04:45 2006 From: lobais at gmail.com (Thomas Dybdahl Ahle) Date: Sat, 16 Dec 2006 17:04:45 +0100 Subject: [Distutils] Error when trying to create an rpm Message-ID: <1166285086.3305.5.camel@localhost.localdomain> Hi, When I try to create an rpm using distutils, I get an error like this: error: Installed (but unpackaged) file(s) found: /usr/share/games/pychess/sidepanel/bookPanel.pyc /usr/share/games/pychess/sidepanel/bookPanel.pyo ... I found this solution http://www.mail-archive.com/distutils-sig at python.org/msg02655.html which removed most of the lines, but besides my modules, I do also have some .py files in a more plug-in like way, which are not stored in modules. For these files the error still appears. I was wondering if there was a way to not have the files created, but simply not have them checked? I don't understand why it does that. From matt at matt-good.net Sun Dec 17 02:34:11 2006 From: matt at matt-good.net (Matt Good) Date: Sat, 16 Dec 2006 20:34:11 -0500 Subject: [Distutils] ORed requirements Proof of Concept Message-ID: <1166319251.19894.18.camel@nny> Based on the recent discussions regarding setuptools requirements where more than one package may satisfy a requirement. Setuptools doesn't have a way to say "my package requires either A or B". I've created an inital patch to the setuptools trunk for specifying this as: install_requires = [ "A | B", ] Requirements with multiple candidates like "A | B" will be scanned first for an already installed version, and if one isn't found it will try to install each candidate in order until one is successful. This patch changes the values returned by pkg_resources.parse_requirements. The items yielded by the method will still be Requirement objects for standard requirements, but it will yield a tuple of Requirement objects if there are multiple candidates. It also does not yet handle this scenario ideally: install_requires = [ 'A | B', ], extras_require = { 'foo': 'B', } If the user wants to install the entry point "foo", setuptools should detect that 'B' will be installed which will satisfy the installation requirement 'A | B'. Right now it will still install 'A' to satisfy the installation requirement, then also install 'B' to satisfy the 'foo' extra. -- Matt Good Index: setuptools/package_index.py =================================================================== --- setuptools/package_index.py (revision 53031) +++ setuptools/package_index.py (working copy) @@ -468,6 +468,13 @@ set, development and system eggs (i.e., those using the ``.egg-info`` format) will be ignored. """ + if isinstance(requirement, tuple): + for req in requirement: + dist = self.fetch_distribution(req, tmpdir, force_scan, source, + develop_ok) + if dist is not None: + return dist + return None # process a Requirement self.info("Searching for %s", requirement) Index: setuptools/command/easy_install.py =================================================================== --- setuptools/command/easy_install.py (revision 53031) +++ setuptools/command/easy_install.py (working copy) @@ -414,7 +414,7 @@ if not self.editable: self.install_site_py() try: - if not isinstance(spec,Requirement): + if not isinstance(spec,(Requirement,tuple)): if URL_SCHEME(spec): # It's a url, download it to tmpdir and process self.not_editable(spec) Index: pkg_resources.py =================================================================== --- pkg_resources.py (revision 53031) +++ pkg_resources.py (working copy) @@ -14,6 +14,7 @@ """ import sys, os, zipimport, time, re, imp, new, pkgutil # XXX +from distutils.errors import DistutilsError from sets import ImmutableSet from os import utime, rename, unlink # capture these to bypass sandboxing from os import open as os_open @@ -465,20 +466,20 @@ processed = {} # set of processed requirements best = {} # key -> dist to_activate = [] + env = [env] - while requirements: - req = requirements.pop(0) # process dependencies breadth-first + def process(req, installer): if req in processed: # Ignore cyclic or redundant dependencies - continue + return dist = best.get(req.key) if dist is None: # Find the best distribution and add it to the map dist = self.by_key.get(req.key) if dist is None: - if env is None: - env = Environment(self.entries) - dist = best[req.key] = env.best_match(req, self, installer) + if env[0] is None: + env[0] = Environment(self.entries) + dist = best[req.key] = env[0].best_match(req, self, installer) if dist is None: raise DistributionNotFound(req) # XXX put more info here to_activate.append(dist) @@ -488,6 +489,27 @@ requirements.extend(dist.requires(req.extras)[::-1]) processed[req] = True + while requirements: + req = requirements.pop(0) # process dependencies breadth-first + if isinstance(req, tuple): + found_match = False + # first try without installer to check pre-installed packages + for inst in (None, installer): + for r in req: + try: + process(r, inst) + found_match = True + break + except (DistributionNotFound, VersionConflict, + DistutilsError), e: + pass + if found_match: + break + if not found_match: + raise e + else: + process(req, installer) + return to_activate # return list of distros to activate def find_plugins(self, @@ -1645,6 +1667,8 @@ s = s.strip() if s and not s.startswith('#'): # skip blank lines/comments yield s + elif isinstance(strs,tuple): + yield ' | '.join(strs) else: for ss in strs: for s in yield_lines(ss): @@ -2212,7 +2236,7 @@ if match: p = match.end() # skip the terminator, if any return line, p, items - for line in lines: + def parse_one(line): match = DISTRO(line) if not match: raise ValueError("Missing distribution spec", line) @@ -2229,9 +2253,16 @@ line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec") specs = [(op,safe_version(val)) for op,val in specs] - yield Requirement(project_name, specs, extras) + return Requirement(project_name, specs, extras) + for line in lines: + req = [parse_one(i) for i in line.split('|')] + if len(req)==1: + yield req[0] + else: + yield tuple(req) + def _sort_dists(dists): tmp = [(dist.hashcmp,dist) for dist in dists] tmp.sort() From ianb at colorstudy.com Mon Dec 18 00:56:22 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 17 Dec 2006 17:56:22 -0600 Subject: [Distutils] workingenv: always installs setuptools? In-Reply-To: <91d4ce220612120657m327d8f25y818999260fa59228@mail.gmail.com> References: <91d4ce220612120657m327d8f25y818999260fa59228@mail.gmail.com> Message-ID: <4585D926.7030105@colorstudy.com> Jean Jordaan wrote: > Hi there > > In install_setuptools I read: "(only called if setuptools is not > installed on the system)". > However, the only call to install_setuptools is unconditional, and it > wants to install a local copy of setuptools although I have it > installed. Which is wrong, the docstring or the code? Probably the docstring, as I found the conditional installation of setuptools problematic and specifically removed it (setuptools is always installed); I guess I forgot to update the docstring at the same time. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From rik at inse.ru Tue Dec 19 15:05:17 2006 From: rik at inse.ru (Roman Kurakin) Date: Tue, 19 Dec 2006 17:05:17 +0300 Subject: [Distutils] configuration files preservation Message-ID: <4587F19D.8070201@inse.ru> While working on one project I've met a problem that I can't to specify to not touch my configuration files for python package that uses setup.py technique. To solve this problem I've implemented the following solution that works for me. If this idea worths it, I can make a patch relative needed version of python. Here is the patch I've done for that project (inlined here to show the idea). # HG changeset patch # Node ID 286864b5c6a30d0ebba5fc99876013c85c06ab59 # Parent fc4dfd9ad64f477a0326995bea7cda12e33ddb39 Add 'preserve files' functionality to setup() while installation. (clean version) diff -r fc4dfd9ad64f -r 286864b5c6a3 setup.py --- a/setup.py Wed Dec 13 12:50:46 2006 +0300 +++ b/setup.py Mon Dec 18 14:01:34 2006 +0300 @@ -26,17 +26,66 @@ # POSSIBILITY OF SUCH DAMAGE. from distutils.core import setup +from distutils.command.install_data import install_data +from distutils.util import change_root, convert_path +from types import StringType -import sys +import sys,os pyver = "%d.%d" % sys.version_info[0:2] + +class smart_install_data(install_data): + def run(self): + # Use revers order for safe removal + for i in range(len(self.data_files)-1, -1, -1): + f = self.data_files[i] + if type(f) is StringType: + continue + if len(f) <= 2: + continue + # Ok, we have additional value, do some magick according it. + if f[2] != "preserve": + # Nope, we do not know it. Just ignore for now. + # Should we scream about it? + continue + # Check a tuple with path to install to and a list of files + dir = convert_path(f[0]) + if not os.path.isabs(dir): + dir = os.path.join(self.install_dir, dir) + elif self.root: + dir = change_root(self.root, dir) + + if f[1] == []: + # If there are no files listed, the user must be + # trying to create an empty directory, so just ignore + # it. + # Should we scream in this case? + continue + + # Check files one by one. + # Use revers order for safe removal + for j in range(len(f[1])-1, -1, -1): + data=f[1][j] + data = convert_path(data) + if not os.path.isfile(data): + # Again skip dirs. + continue + dst = os.path.join(dir, os.path.basename(data)) + if not os.path.exists(dst): + continue + del f[1][j] + if len(f[1]) == 0: + del self.data_files[i] + + return install_data.run(self) setup(name = 'usher', version = '0.1', package_dir = { 'usher':'python' }, packages = ['usher', 'usher.ushercli', 'usher.usherlnm', 'usher.utils', 'usher.usherctrl'], + cmdclass = {'install_data':smart_install_data}, data_files = [ ('/etc/init.d', ['initscripts/usherctrl', 'initscripts/usherlnm']), - ('/etc/usher', ['configs/usherctrl.config', 'configs/usherlnm.config', 'configs/ushercli.config']), + ('/etc/usher', ['configs/usherctrl.config', 'configs/usherlnm.config', 'configs/ushercli.config'], "preserve"), ("/usr/lib/python%s/site-packages/usher/usherctrl" % pyver, ['python/usherctrl/app.tac']), ("/usr/lib/python%s/site-packages/usher/usherlnm" % pyver, ['python/usherlnm/app.tac']) ] From jim at zope.com Fri Dec 22 22:03:06 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 22 Dec 2006 16:03:06 -0500 Subject: [Distutils] zc.buildout minitutorial at PyCon Message-ID: <458C480A.7090301@zope.com> I'm giving a pair of mini tutorials at PyCon as a single tutorial: http://us.pycon.org/TX2007/TutorialsAM#AM7 One of these is on zc.buildout. This will be a good opportunity to learn about zc.buildout. It will also be a good opportunity to get an introduction to creating eggs. I've included the outline for the buildout mini tutorial below. You can sign up for the tutorial when you register for PyCon: http://us.pycon.org/TX2007/Registration The other mini tutorial is on the Zope Component Architecture. It is aimed a Python programmers in general. You needed be or intend to be a Zope developer to benefit from it. Jim Introduction to zc.buildout - Why zc.buildout? - System assembly - why not make or scons? - Python application assembly with eggs - Support for developers - Assembling applications versus installing packages - Why not easy_install? - A Python Egg Primer Here I'll provide an introduction to using eggs from the point of view of a the lazy programmer. I'll omit setuptools arcana and focus on the core concepts that make eggs so useful. - Eggs They can sound complicated, but they are actually pretty simple. - easy_install and setuptools - Some egg-related jargon - distributions - source and binary distributions - eggs and develop eggs - requirements - index- and link-servers - entry points and script generation - Quick buildout introduction - A buildout.cfg file - Parts - Recipes - Special recipes for supporting Python application development - Buildout examples - Using buildout to try out new packages - Using buildout to create a development environment - Installing egg-based scripts for personal use - System assembly - A lazy programmer's introduction to creating eggs - Develop eggs, a minimal starting point - Adding data needed for distribution - Creating distribution files - Extending buildout with new recipes - The recipe API - A sample recipe - Part dependencies - Advanced topics - Sharing an egg repository across projects - Offline mode - Multiple deployment environments (dev/stage/prod) and multiple machines. - Using buildout with system-packaging tools (e.g. RPM). -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Fri Dec 22 22:47:23 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 22 Dec 2006 16:47:23 -0500 Subject: [Distutils] zc.buildout minitutorial at PyCon In-Reply-To: <458C480A.7090301@zope.com> References: <458C480A.7090301@zope.com> Message-ID: <458C526B.8060907@zope.com> Jim Fulton wrote: > The other mini tutorial is on the Zope Component Architecture. > It is aimed a Python programmers in general. You needed be > or intend to be a Zope developer to benefit from it. I tried to type "needn't". Sorry. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From davidsmith at acm.org Thu Dec 28 05:25:34 2006 From: davidsmith at acm.org (David Smith) Date: Thu, 28 Dec 2006 13:25:34 +0900 Subject: [Distutils] HTTP authentication support Message-ID: <87hcvglk8x.fsf@exponent.kumachan.no-ip.org> Index: setuptools/package_index.py =================================================================== --- setuptools/package_index.py (????? 53164) +++ setuptools/package_index.py (?????) @@ -20,9 +20,27 @@ __all__ = [ 'PackageIndex', 'distros_for_url', 'parse_bdist_wininst', - 'interpret_distro_name', + 'interpret_distro_name', 'detect_auth_scheme', ] +def detect_auth_scheme(url): + """Return (scheme, realm) or (None, None) for a URL""" + scheme, realm = None, None + try: + f = urllib2.urlopen(url) + f.close() + except urllib2.HTTPError, e: + if e.code == 401: + import re + realm = re.findall('realm="([^"]*)"', e.headers['WWW-Authenticate']) + if realm: + realm = realm[0] + else: + realm = None + scheme = e.headers['WWW-Authenticate'].split()[0].lower() + else: + raise e + return (scheme, realm) def parse_bdist_wininst(name): """Return (base,pyversion) or (None,None) for possible .exe name""" @@ -166,7 +184,7 @@ """A distribution index that scans web pages for download URLs""" def __init__(self, index_url="http://www.python.org/pypi", hosts=('*',), - sf_mirrors=None, *args, **kw + sf_mirrors=None, username=None, password=None, *args, **kw ): Environment.__init__(self,*args,**kw) self.index_url = index_url + "/"[:not index_url.endswith('/')] @@ -182,6 +200,8 @@ self.sf_mirrors = map(str.strip, sf_mirrors) else: self.sf_mirrors = () + self.username = username + self.password = password def _get_mirrors(self): @@ -654,13 +674,42 @@ - def open_url(self, url): + def open_url(self, url, username=None, password=None): + """username and/or password can be a string or a callable. + If a callable, they are called with the host, scheme, and realm.""" + + username = username or self.username + password = password or self.password + if url.startswith('file:'): return local_open(url) try: + scheme, realm = detect_auth_scheme(url) + host = "" + if isinstance(url, urllib2.Request): + host = url.get_host() + else: + host = urlparse.urlparse(url)[1] + if not scheme: + return urllib2.urlopen(url) + else: + password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() + if scheme.lower() == 'digest': + auth = urllib2.HTTPDigestAuthHandler(password_mgr) + elif scheme.lower() == 'basic': + auth = urllib2.HTTPBasicAuthHandler(password_mgr) + else: + auth = None + if auth: + if callable(username): + username = username(host, scheme, realm) + if callable(password): + password = password(host, scheme, realm) + password_mgr.add_password(realm, host, username, password) + opener = urllib2.build_opener(auth) request = urllib2.Request(url) request.add_header('User-Agent', user_agent) - return urllib2.urlopen(request) + return opener.open(request) except urllib2.HTTPError, v: return v except urllib2.URLError, v: Index: setuptools/command/easy_install.py =================================================================== --- setuptools/command/easy_install.py (????? 53164) +++ setuptools/command/easy_install.py (?????) @@ -9,7 +9,8 @@ __ http://peak.telecommunity.com/DevCenter/EasyInstall """ -import sys, os.path, zipimport, shutil, tempfile, zipfile, re, stat, random +import sys, os.path, zipimport, shutil, tempfile, zipfile, re, stat, \ + random, getpass from glob import glob from setuptools import Command from setuptools.sandbox import run_setup @@ -71,6 +72,8 @@ ('no-deps', 'N', "don't install dependencies"), ('allow-hosts=', 'H', "pattern(s) that hostnames must match"), ('sf-mirrors=', None, "Sourceforge mirror(s) to use"), + ('username', 'u', "username for HTTP authentication"), + ('password', 'p', "password for HTTP authentication"), ] boolean_options = [ 'zip-ok', 'multi-version', 'exclude-scripts', 'upgrade', 'always-copy', @@ -110,7 +113,17 @@ self.distribution._set_command_options( self, self.distribution.get_option_dict('easy_install') ) + self.username = self.prompt_username + self.password = self.prompt_password + def prompt_username(self, host, scheme, realm): + self.announce("Host %s requires %s authentication." % (host, scheme), + level=9) + return raw_input("Username for %s@%s: " % (realm, host)) + + def prompt_password(self, host, scheme, realm): + return getpass.getpass("Password for %s@%s: " % (realm, host)) + def delete_blockers(self, blockers): for filename in blockers: if os.path.exists(filename) or os.path.islink(filename): @@ -169,7 +182,8 @@ if self.package_index is None: self.package_index = self.create_index( self.index_url, search_path = self.shadow_path, hosts=hosts, - sf_mirrors = self.sf_mirrors + sf_mirrors = self.sf_mirrors, username=self.username, + password=self.password ) self.local_index = Environment(self.shadow_path+sys.path) From davidsmith at acm.org Thu Dec 28 05:25:50 2006 From: davidsmith at acm.org (David Smith) Date: Thu, 28 Dec 2006 13:25:50 +0900 Subject: [Distutils] HTTP authentication support Message-ID: <87fyb0lk8h.fsf@exponent.kumachan.no-ip.org> Index: setuptools/package_index.py =================================================================== --- setuptools/package_index.py (????? 53164) +++ setuptools/package_index.py (?????) @@ -20,9 +20,27 @@ __all__ = [ 'PackageIndex', 'distros_for_url', 'parse_bdist_wininst', - 'interpret_distro_name', + 'interpret_distro_name', 'detect_auth_scheme', ] +def detect_auth_scheme(url): + """Return (scheme, realm) or (None, None) for a URL""" + scheme, realm = None, None + try: + f = urllib2.urlopen(url) + f.close() + except urllib2.HTTPError, e: + if e.code == 401: + import re + realm = re.findall('realm="([^"]*)"', e.headers['WWW-Authenticate']) + if realm: + realm = realm[0] + else: + realm = None + scheme = e.headers['WWW-Authenticate'].split()[0].lower() + else: + raise e + return (scheme, realm) def parse_bdist_wininst(name): """Return (base,pyversion) or (None,None) for possible .exe name""" @@ -166,7 +184,7 @@ """A distribution index that scans web pages for download URLs""" def __init__(self, index_url="http://www.python.org/pypi", hosts=('*',), - sf_mirrors=None, *args, **kw + sf_mirrors=None, username=None, password=None, *args, **kw ): Environment.__init__(self,*args,**kw) self.index_url = index_url + "/"[:not index_url.endswith('/')] @@ -182,6 +200,8 @@ self.sf_mirrors = map(str.strip, sf_mirrors) else: self.sf_mirrors = () + self.username = username + self.password = password def _get_mirrors(self): @@ -654,13 +674,42 @@ - def open_url(self, url): + def open_url(self, url, username=None, password=None): + """username and/or password can be a string or a callable. + If a callable, they are called with the host, scheme, and realm.""" + + username = username or self.username + password = password or self.password + if url.startswith('file:'): return local_open(url) try: + scheme, realm = detect_auth_scheme(url) + host = "" + if isinstance(url, urllib2.Request): + host = url.get_host() + else: + host = urlparse.urlparse(url)[1] + if not scheme: + return urllib2.urlopen(url) + else: + password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() + if scheme.lower() == 'digest': + auth = urllib2.HTTPDigestAuthHandler(password_mgr) + elif scheme.lower() == 'basic': + auth = urllib2.HTTPBasicAuthHandler(password_mgr) + else: + auth = None + if auth: + if callable(username): + username = username(host, scheme, realm) + if callable(password): + password = password(host, scheme, realm) + password_mgr.add_password(realm, host, username, password) + opener = urllib2.build_opener(auth) request = urllib2.Request(url) request.add_header('User-Agent', user_agent) - return urllib2.urlopen(request) + return opener.open(request) except urllib2.HTTPError, v: return v except urllib2.URLError, v: Index: setuptools/command/easy_install.py =================================================================== --- setuptools/command/easy_install.py (????? 53164) +++ setuptools/command/easy_install.py (?????) @@ -9,7 +9,8 @@ __ http://peak.telecommunity.com/DevCenter/EasyInstall """ -import sys, os.path, zipimport, shutil, tempfile, zipfile, re, stat, random +import sys, os.path, zipimport, shutil, tempfile, zipfile, re, stat, \ + random, getpass from glob import glob from setuptools import Command from setuptools.sandbox import run_setup @@ -71,6 +72,8 @@ ('no-deps', 'N', "don't install dependencies"), ('allow-hosts=', 'H', "pattern(s) that hostnames must match"), ('sf-mirrors=', None, "Sourceforge mirror(s) to use"), + ('username', 'u', "username for HTTP authentication"), + ('password', 'p', "password for HTTP authentication"), ] boolean_options = [ 'zip-ok', 'multi-version', 'exclude-scripts', 'upgrade', 'always-copy', @@ -110,7 +113,17 @@ self.distribution._set_command_options( self, self.distribution.get_option_dict('easy_install') ) + self.username = self.prompt_username + self.password = self.prompt_password + def prompt_username(self, host, scheme, realm): + self.announce("Host %s requires %s authentication." % (host, scheme), + level=9) + return raw_input("Username for %s@%s: " % (realm, host)) + + def prompt_password(self, host, scheme, realm): + return getpass.getpass("Password for %s@%s: " % (realm, host)) + def delete_blockers(self, blockers): for filename in blockers: if os.path.exists(filename) or os.path.islink(filename): @@ -169,7 +182,8 @@ if self.package_index is None: self.package_index = self.create_index( self.index_url, search_path = self.shadow_path, hosts=hosts, - sf_mirrors = self.sf_mirrors + sf_mirrors = self.sf_mirrors, username=self.username, + password=self.password ) self.local_index = Environment(self.shadow_path+sys.path) From pje at telecommunity.com Thu Dec 28 06:45:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Dec 2006 00:45:15 -0500 Subject: [Distutils] HTTP authentication support In-Reply-To: <87fyb0lk8h.fsf@exponent.kumachan.no-ip.org> Message-ID: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> At 01:25 PM 12/28/2006 +0900, David Smith wrote: >Hello, > >I needed support in easy_install for installing from HTTP >digest and basic authenticated sites so I wrote the included >patch. It currently supports specifying username and password >information at the command line (however misguided that may be) >and prompting the user interactively. It should also be easy to >integrate with graphical interfaces. Please consider it for >inclusion in the next release. Thanks for the submission, but for a number of reasons, I don't think it makes sense to put HTTP authentication directly into easy_install as a core feature. For version 0.7 and beyond, however, there will probably be some sort of plugin mechanism to enable alternative file transport protocols, so you should be able to plug this sort of thing in without patching easy_install in that case. From davidsmith at acm.org Thu Dec 28 07:12:08 2006 From: davidsmith at acm.org (David Smith) Date: Thu, 28 Dec 2006 15:12:08 +0900 Subject: [Distutils] HTTP authentication support In-Reply-To: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> (Phillip J. Eby's message of "Thu\, 28 Dec 2006 00\:45\:15 -0500") References: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> Message-ID: <87tzzgk0qv.fsf@exponent.kumachan.no-ip.org> A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20061228/813225d9/attachment.pgp From pje at telecommunity.com Thu Dec 28 07:38:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Dec 2006 01:38:05 -0500 Subject: [Distutils] HTTP authentication support In-Reply-To: <87tzzgk0qv.fsf@exponent.kumachan.no-ip.org> References: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> At 03:12 PM 12/28/2006 +0900, David Smith wrote: >The usecase is too obvious: using setuptools for >distribution of modules that are not publically available, such >as commercial or pre-release development. If they're not publically available, why use authenticated HTTP at all? Among the many alternatives that work with easy_install today are: * rsync-mirrored directories * NFS-mounted directories * ssh-transport SVN servers (you even get prompted for your credentials!) * proxied HTTP And in a lot of cases, one or more of the above is either already set up or easier to add than a password-protected web directory. It seems to me that the only case where HTTP auth helps is if you were already distributing your files that way. >Until 0.7 is released, considering the very small size of the >patch, could we add this for now and refactor it later? 0.6 is in a bugfix-only mode at the moment. It *may* get some extra features in the bootstrapping system (ez_setup) before the final release, but that's basically it. Please note that if you want to add this feature, nothing is stopping you from subclassing PackageIndex and creating an easy_install() instance that uses an instance of your subclass. This can be done without any patching; you would simply do something like this:: class MyPackageIndex(PackageIndex): # ... class my_easy_install(easy_install): create_index = MyPackageIndex # makes it use your class instead # ... And then either write your own version of the 'main()' function, or monkeypatch your class in. Add a setup.py to your little package to specify an entry point, and your easy_install command-line replacement is good to go. If your main concern is for your projects that use setuptools, you don't even need the main() function. Just specify: cmdclass = dict(easy_install=my_easy_install), in your setup() call to tell setuptools to use your replacement class. In other words, you don't need to have this patch accepted in order to integrate the functionality with the existing system. It's just not a nice orthogonal plugin, as you have to supply replacement subclasses in the normal distutils style, instead of entry points in the setuptools style. From davidsmith at acm.org Thu Dec 28 07:55:41 2006 From: davidsmith at acm.org (David Smith) Date: Thu, 28 Dec 2006 15:55:41 +0900 Subject: [Distutils] HTTP authentication support In-Reply-To: <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> (Phillip J. Eby's message of "Thu\, 28 Dec 2006 01\:38\:05 -0500") References: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> Message-ID: <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> "Phillip J. Eby" writes: > At 03:12 PM 12/28/2006 +0900, David Smith wrote: >>The usecase is too obvious: using setuptools for >>distribution of modules that are not publically available, such >>as commercial or pre-release development. > > If they're not publically available, why use authenticated HTTP > at all? Among the many alternatives that work with > easy_install today are: > > * rsync-mirrored directories > * NFS-mounted directories > * ssh-transport SVN servers (you even get prompted for your credentials!) > * proxied HTTP I believe HTTP authenticated subversion is more common than all of the above. Additionally, the proxied HTTP support also lacks authentication support, which the patch rectifies. > It seems to me that the only case where HTTP auth helps is if > you were already distributing your files that way. I think you'd agree that HTTP is by far the most standard distribution method than any of the others currently supported by setuptools. We're really not talking about a special case. > In other words, you don't need to have this patch accepted in > order to integrate the functionality with the existing system. I admit it's easy enough to fix locally; that's what I had been doing. The problem I have with that is it's not a special case and if you support ssl-encrypted HTTP, you should also support HTTP authentication considering how the two are often used together. Thanks, -- David D. Smith -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20061228/b1bdae51/attachment-0001.pgp From davidsmith at acm.org Thu Dec 28 08:07:03 2006 From: davidsmith at acm.org (David Smith) Date: Thu, 28 Dec 2006 16:07:03 +0900 Subject: [Distutils] HTTP authentication support In-Reply-To: <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> (David Smith's message of "Thu\, 28 Dec 2006 15\:55\:41 +0900") References: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> Message-ID: <87bqlojy7c.fsf@exponent.kumachan.no-ip.org> I forgot to mention that I am not seeking this to get included in the stable 0.6 branch. Apply to current 0.7 SVN and refactor later is what I wanted to say. -- David D. Smith -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20061228/5608fbb0/attachment.pgp From rik at inse.ru Thu Dec 28 11:06:55 2006 From: rik at inse.ru (Roman Kurakin) Date: Thu, 28 Dec 2006 13:06:55 +0300 Subject: [Distutils] configuration files preservation In-Reply-To: <4587F19D.8070201@inse.ru> References: <4587F19D.8070201@inse.ru> Message-ID: <4593973F.5010009@inse.ru> Any comments (keepalivemessage)? rik Roman Kurakin: > While working on one project I've met a problem that I can't to specify to > not touch my configuration files for python package that uses setup.py > technique. To solve this problem I've implemented the following solution > that works for me. If this idea worths it, I can make a patch relative > needed > version of python. > > Here is the patch I've done for that project (inlined here to show the > idea). > > > # HG changeset patch > # Node ID 286864b5c6a30d0ebba5fc99876013c85c06ab59 > # Parent fc4dfd9ad64f477a0326995bea7cda12e33ddb39 > Add 'preserve files' functionality to setup() while installation. (clean > version) > > diff -r fc4dfd9ad64f -r 286864b5c6a3 setup.py > --- a/setup.py Wed Dec 13 12:50:46 2006 +0300 > +++ b/setup.py Mon Dec 18 14:01:34 2006 +0300 > @@ -26,17 +26,66 @@ > # POSSIBILITY OF SUCH DAMAGE. > > from distutils.core import setup > +from distutils.command.install_data import install_data > +from distutils.util import change_root, convert_path > +from types import StringType > > -import sys > +import sys,os > pyver = "%d.%d" % sys.version_info[0:2] > + > +class smart_install_data(install_data): > + def run(self): > + # Use revers order for safe removal > + for i in range(len(self.data_files)-1, -1, -1): > + f = self.data_files[i] > + if type(f) is StringType: > + continue > + if len(f) <= 2: > + continue > + # Ok, we have additional value, do some magick according it. > + if f[2] != "preserve": > + # Nope, we do not know it. Just ignore for now. > + # Should we scream about it? > + continue > + # Check a tuple with path to install to and a list of files > + dir = convert_path(f[0]) > + if not os.path.isabs(dir): > + dir = os.path.join(self.install_dir, dir) > + elif self.root: > + dir = change_root(self.root, dir) > + > + if f[1] == []: > + # If there are no files listed, the user must be > + # trying to create an empty directory, so just ignore > + # it. > + # Should we scream in this case? > + continue > + > + # Check files one by one. > + # Use revers order for safe removal > + for j in range(len(f[1])-1, -1, -1): > + data=f[1][j] > + data = convert_path(data) > + if not os.path.isfile(data): > + # Again skip dirs. > + continue > + dst = os.path.join(dir, os.path.basename(data)) > + if not os.path.exists(dst): > + continue > + del f[1][j] > + if len(f[1]) == 0: > + del self.data_files[i] > + > + return install_data.run(self) > > setup(name = 'usher', > version = '0.1', > package_dir = { 'usher':'python' }, > packages = ['usher', 'usher.ushercli', 'usher.usherlnm', > 'usher.utils', 'usher.usherctrl'], > + cmdclass = {'install_data':smart_install_data}, > data_files = [ > ('/etc/init.d', ['initscripts/usherctrl', > 'initscripts/usherlnm']), > - ('/etc/usher', ['configs/usherctrl.config', > 'configs/usherlnm.config', 'configs/ushercli.config']), > + ('/etc/usher', ['configs/usherctrl.config', > 'configs/usherlnm.config', 'configs/ushercli.config'], "preserve"), > ("/usr/lib/python%s/site-packages/usher/usherctrl" % pyver, > ['python/usherctrl/app.tac']), > ("/usr/lib/python%s/site-packages/usher/usherlnm" % pyver, > ['python/usherlnm/app.tac']) > ] > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From pje at telecommunity.com Thu Dec 28 17:20:00 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Dec 2006 11:20:00 -0500 Subject: [Distutils] configuration files preservation In-Reply-To: <4593973F.5010009@inse.ru> References: <4587F19D.8070201@inse.ru> <4587F19D.8070201@inse.ru> Message-ID: <5.1.1.6.0.20061228111757.026b4108@sparrow.telecommunity.com> At 01:06 PM 12/28/2006 +0300, Roman Kurakin wrote: >Any comments (keepalivemessage)? Unfortunately, the distutils aren't really actively maintained by anyone (except to do bug fixes), which is probably why you haven't received any reply. From bhendrix at enthought.com Thu Dec 28 17:23:35 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 28 Dec 2006 10:23:35 -0600 Subject: [Distutils] HTTP authentication support In-Reply-To: <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> References: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> Message-ID: <4593EF87.5010407@enthought.com> We'd really like to see authenticated HTTP as well. In our case we have eggs which are not publicly available, but would like to distribute them to our customers. Bryce David Smith wrote: > "Phillip J. Eby" writes: > > >> At 03:12 PM 12/28/2006 +0900, David Smith wrote: >> >>> The usecase is too obvious: using setuptools for >>> distribution of modules that are not publically available, such >>> as commercial or pre-release development. >>> >> If they're not publically available, why use authenticated HTTP >> at all? Among the many alternatives that work with >> easy_install today are: >> >> * rsync-mirrored directories >> * NFS-mounted directories >> * ssh-transport SVN servers (you even get prompted for your credentials!) >> * proxied HTTP >> > > I believe HTTP authenticated subversion is more common than all > of the above. Additionally, the proxied HTTP support also lacks > authentication support, which the patch rectifies. > > >> It seems to me that the only case where HTTP auth helps is if >> you were already distributing your files that way. >> > > I think you'd agree that HTTP is by far the most standard > distribution method than any of the others currently supported > by setuptools. We're really not talking about a special case. > > >> In other words, you don't need to have this patch accepted in >> order to integrate the functionality with the existing system. >> > > I admit it's easy enough to fix locally; that's what I had been > doing. The problem I have with that is it's not a special case > and if you support ssl-encrypted HTTP, you should also support > HTTP authentication considering how the two are often used > together. > > Thanks, > > ------------------------------------------------------------------------ > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20061228/3e42fe4d/attachment.htm From rik at inse.ru Thu Dec 28 22:59:40 2006 From: rik at inse.ru (Roman Kurakin) Date: Fri, 29 Dec 2006 00:59:40 +0300 Subject: [Distutils] configuration files preservation In-Reply-To: <5.1.1.6.0.20061228111757.026b4108@sparrow.telecommunity.com> References: <4587F19D.8070201@inse.ru> <4587F19D.8070201@inse.ru> <5.1.1.6.0.20061228111757.026b4108@sparrow.telecommunity.com> Message-ID: <45943E4C.40705@inse.ru> Phillip J. Eby: > At 01:06 PM 12/28/2006 +0300, Roman Kurakin wrote: >> Any comments (keepalivemessage)? > > Unfortunately, the distutils aren't really actively maintained by anyone > (except to do bug fixes), which is probably why you haven't received > any reply. How should I treat your answer? Eq, if I'll make a full patch (with doc changes), post it here and will not get any answer, is there any chances this it will be accepted on SF, or it is better to not lose the time? Thanks for reply any way. :-) rik From pje at telecommunity.com Thu Dec 28 23:49:53 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Dec 2006 17:49:53 -0500 Subject: [Distutils] configuration files preservation In-Reply-To: <45943E4C.40705@inse.ru> References: <5.1.1.6.0.20061228111757.026b4108@sparrow.telecommunity.com> <4587F19D.8070201@inse.ru> <4587F19D.8070201@inse.ru> <5.1.1.6.0.20061228111757.026b4108@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061228174830.02946c28@sparrow.telecommunity.com> At 12:59 AM 12/29/2006 +0300, Roman Kurakin wrote: >Phillip J. Eby: >>At 01:06 PM 12/28/2006 +0300, Roman Kurakin wrote: >>>Any comments (keepalivemessage)? >> >>Unfortunately, the distutils aren't really actively maintained by anyone >>(except to do bug fixes), which is probably why you haven't received any >>reply. >How should I treat your answer? Eq, if I'll make a full patch (with doc >changes), >post it here and will not get any answer, is there any chances this it >will be >accepted on SF, or it is better to not lose the time? I think that to have a chance, you'd need to at least include appropriate documentation patches. From jim at zope.com Fri Dec 29 14:41:10 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 29 Dec 2006 08:41:10 -0500 Subject: [Distutils] HTTP authentication support In-Reply-To: <87hcvglk8x.fsf@exponent.kumachan.no-ip.org> References: <87hcvglk8x.fsf@exponent.kumachan.no-ip.org> Message-ID: <9F26AF31-2B40-495C-95C0-B6399495264F@zope.com> On Dec 27, 2006, at 11:25 PM, David Smith wrote: > I needed support in easy_install for installing from HTTP > digest and basic authenticated sites so I wrote the included > patch. It currently supports specifying username and password > information at the command line (however misguided that may be) > and prompting the user interactively. It should also be easy to > integrate with graphical interfaces. Please consider it for > inclusion in the next release. I don't know if it is an option for you to use zc.buildout. If it is, then zc.buildout has an extension that lets you use sftp: http://www.python.org/pypi/zc.buildoutsftp This has the advantage that, if you use an SSH agent, you don't have to enter credentials when you download a transaction. And, of course, it is more secure than digest or basic authentication. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Dec 30 17:16:04 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 30 Dec 2006 11:16:04 -0500 Subject: [Distutils] HTTP authentication support In-Reply-To: <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> References: <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20061230103134.02a8b8b8@sparrow.telecommunity.com> At 03:55 PM 12/28/2006 +0900, David Smith wrote: >"Phillip J. Eby" writes: > > > At 03:12 PM 12/28/2006 +0900, David Smith wrote: > >>The usecase is too obvious: using setuptools for > >>distribution of modules that are not publically available, such > >>as commercial or pre-release development. > > > > If they're not publically available, why use authenticated HTTP > > at all? Among the many alternatives that work with > > easy_install today are: > > > > * rsync-mirrored directories > > * NFS-mounted directories > > * ssh-transport SVN servers (you even get prompted for your credentials!) > > * proxied HTTP Also, I forgot to mention password-protected FTP, which easy_install also supports (using ftp://user:pass at host URLs). >I believe HTTP authenticated subversion is more common than all >of the above. I've never set up svn-over-http before, as I find it a PITA compared to plain svn or svn-over-ssh. I guess tastes vary in this respect. > Additionally, the proxied HTTP support also lacks >authentication support, which the patch rectifies. It's relatively easy to add support for Basic auth with credentialed HTTP URLs, so that's what I've done. I'd rather not make easy_install interactive at this time, nor add one-shot user/password command line options, nor make the behavior dependent on the server's response. Using URLs with credentials also allows more than one set to be specified on the command line (or via dependency_links). The patch is minor enough that I added it to 0.6; it only affects URLs that would have caused today's easy_install to crash with an httplib.InvalidURL error, anyway. The easiest way to use it is to use a -f or dependency_links URL that points to a page with links to the actual downloads. As long as they are the same scheme and host, they'll inherit the credentials from the original URL.