From jorge.vargas at gmail.com Sat Jul 1 05:39:22 2006 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Fri, 30 Jun 2006 23:39:22 -0400 Subject: [Distutils] a patch for distutils + gcc + linux, that was never apply, yet will solve many problems. Message-ID: <32822fe60606302039n2246fbfay53a3fed9bf50d94f@mail.gmail.com> Hello everyone I did a package for gentoo (ebuild) a while ago and it has a C/C++ extension, to compile agains a gecko-compliant mozilla product. Long story short our solution (build agains mozilla) seem to had nothing to do with the real bug which someone posted today. It seems there is a bug in distutils itself, because it is passing the wrong arguments to g++, the most incredible part is that the bug is so simple to fix yet there is no reply, maybe this got lost in the main python repository. here is the most important part of his post ---- /usr/lib/python2.4/distutils/unixccompiler.py to build. ** /usr/lib/python2.4/distutils/unixccompiler.py fills the 'CC' executable variable by invoking /usr/lib/python2.4/distutils/sysconfig.py ** /usr/lib/python2.4/distutils/unixccompiler.py contains a condition that says: elif compiler[:3] == "gcc" or compiler[:3] == "g++": return "-Wl,-R" + dir ** /usr/lib/python2.4/distutils/sysconfig.py returns the 'CC' executable variable by reading from /usr/lib/python2.4/config/Makefile ** If your system is like many others, it will return something like 'CC=i686-pc-linux-gnu-gcc -pthread', will never == "gcc" and so the correct linker options will not be used. ---- if anyone could please verify it and maybe find out why it wasn't apply, honestly I have very little knowlegde of C and it's integration with python to be able to verify it. his full post is at http://bugs.gentoo.org/show_bug.cgi?id=131527#c36 the original disutil gentoo bug: http://bugs.gentoo.org/show_bug.cgi?id=138647 the patch at sourceforce: http://sourceforge.net/tracker/index.php?func=detail&aid=1254718&group_id=5470&atid=305470 thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060630/e95bff97/attachment.htm From ronaldoussoren at mac.com Sun Jul 2 10:27:37 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 2 Jul 2006 10:27:37 +0200 Subject: [Distutils] Best practices for creating eggs? In-Reply-To: <5.1.1.6.0.20060629164229.03783d38@sparrow.telecommunity.com> References: <5.1.1.6.0.20060629102724.01ef1868@sparrow.telecommunity.com> <5.1.1.6.0.20060628173425.02b89198@sparrow.telecommunity.com> <5.1.1.6.0.20060629102724.01ef1868@sparrow.telecommunity.com> <5.1.1.6.0.20060629164229.03783d38@sparrow.telecommunity.com> Message-ID: <760F2DBC-5D94-4AE5-9C27-19274C9DB85D@mac.com> On Jun 29, 2006, at 10:43 PM, Phillip J. Eby wrote: > At 04:13 PM 6/29/2006 +0100, Paul Moore wrote: >> Agreed. But in the absence of a standard, supporting package authors' >> existing approaches, which work with other distutils mechanisms, >> seems >> like a reasonable requirement. > > Anything that the package author installs as package data, or using > the > data_files option to setup(), is included in the egg. And if you > install > unzipped, you should be able to browse the included docs. Installation of anything but python modules and the data they need to function seems to be outside of the scope of eggs. There's nothing wrong with that, but just means that there is room for something beyond eggs. As an example why "stuff everything in an egg" can't work for everything, PyObjC uses bdist_mpkg to create an installer that also installs additional templates for Xcode, those cannot be part of an egg because the egg will not be in a location where Xcode will look for templates. AFAIK some future version of bdist_mpkg will install python packages as eggs, while still allowing installation of other stuff. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2157 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060702/d0ab4092/attachment.bin From p.f.moore at gmail.com Sun Jul 2 22:45:39 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 2 Jul 2006 21:45:39 +0100 Subject: [Distutils] Problem between bdist_msi and egg-info (Python 2.5b1) In-Reply-To: <79990c6b0606300234kcebb86epf1717b32dac005a6@mail.gmail.com> References: <5.1.1.6.0.20060629164939.03b38078@sparrow.telecommunity.com> <79990c6b0606300234kcebb86epf1717b32dac005a6@mail.gmail.com> Message-ID: <79990c6b0607021345m517edpb735d4dbaab5b857@mail.gmail.com> I have now submitted a bug (http://www.python.org/sf/1515998). Paul. From pje at telecommunity.com Mon Jul 3 19:05:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 03 Jul 2006 13:05:23 -0400 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <17577.19125.911743.344271@gargle.gargle.HOWL> References: <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> At 06:49 PM 7/3/2006 +0200, Matthias Klose wrote: >Phillip J. Eby writes: > > At 10:28 PM 6/9/2006 +0200, Matthias Klose wrote: > > >Phillip J. Eby writes: > > > > At 09:51 PM 6/9/2006 +0200, Matthias Klose wrote: > > > > >Hi, > > > > > > > > > >when installing an egg in a system-installed python version, then you > > > > >do have another python version information in the egg_info directory > > > > >name (py2.x). Is it possible or advisable to exclude that information > > > > >from the name? At least I know that I'm installing into > > > > >/usr/lib/python2.3/site-packages. > > > > > > > > I don't understand your question. There is no 'egg_info' > directory. Are > > > > you talking about a .egg/EGG-INFO directory or an '.egg-info' > directory? > > > > > >sorry, > > > > > > /usr/lib/python2.3/site-packages/setuptools-0.6b2-py2.3.egg-info > > > > > >would it be safe to rename that to > > > > > > /usr/lib/python2.3/site-packages/setuptools-0.6b2.egg-info > > > > > >without loosing functionality? > > > > Yes. > >what about installing the .egg-info directory without version >information when --single-version-externally-managed is used? Note that the .pyc files will be built for a specific Python version; that's why the version number is there. Not including the version number won't magically make it work with other Python versions. >Another unrelated thing: Debian is supposed to ship the source code >for binaries, but it's not included in the package for cli.exe and >gui.exe. Could you point me to the source code and/or include it in >the next release? It's included in the source release, along with all the documentation. The file you're looking for is 'launcher.c'. Of course, the .exe files aren't used on non-Windows platforms anyway. From doko at cs.tu-berlin.de Tue Jul 4 08:42:05 2006 From: doko at cs.tu-berlin.de (Matthias Klose) Date: Tue, 4 Jul 2006 08:42:05 +0200 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> References: <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> Message-ID: <17578.3517.605579.603127@gargle.gargle.HOWL> Phillip J. Eby writes: > >what about installing the .egg-info directory without version > >information when --single-version-externally-managed is used? > > Note that the .pyc files will be built for a specific Python version; > that's why the version number is there. yes, but this shouldn't be necessary if you install into a path which is known to belong to a specific python version (i.e. /usr/lib/python2.4/site-pacakges) > Not including the version number > won't magically make it work with other Python versions. it will work, but the files will be re-compiled again. > >Another unrelated thing: Debian is supposed to ship the source code > >for binaries, but it's not included in the package for cli.exe and > >gui.exe. Could you point me to the source code and/or include it in > >the next release? > > It's included in the source release, along with all the documentation. The > file you're looking for is 'launcher.c'. Of course, the .exe files aren't > used on non-Windows platforms anyway. at least with distutils I was able to build installers for Windows on Linux. I didn't check if that works with setuptools as well. Matthias From pje at telecommunity.com Tue Jul 4 16:09:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 04 Jul 2006 10:09:10 -0400 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <17578.3517.605579.603127@gargle.gargle.HOWL> References: <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> At 08:42 AM 7/4/2006 +0200, Matthias Klose wrote: >Phillip J. Eby writes: > > >what about installing the .egg-info directory without version > > >information when --single-version-externally-managed is used? > > > > Note that the .pyc files will be built for a specific Python version; > > that's why the version number is there. > >yes, but this shouldn't be necessary if you install into a path which >is known to belong to a specific python version >(i.e. /usr/lib/python2.4/site-pacakges) Setuptools doesn't inspect the path it's installing to in order to know whether it happens to contain the Python version already. :) I don't see what problem you're trying to solve here. > > >Another unrelated thing: Debian is supposed to ship the source code > > >for binaries, but it's not included in the package for cli.exe and > > >gui.exe. Could you point me to the source code and/or include it in > > >the next release? > > > > It's included in the source release, along with all the > documentation. The > > file you're looking for is 'launcher.c'. Of course, the .exe files aren't > > used on non-Windows platforms anyway. > >at least with distutils I was able to build installers for Windows on >Linux. I didn't check if that works with setuptools as well. It should work fine for packages that don't have any scripts. Scripts probably won't work correctly due to platform-specific path issues, and extension modules built for Linux obviously won't work on Windows as they won't even have the right extension in the filename. From int at mtx.ru Wed Jul 5 11:11:53 2006 From: int at mtx.ru (Pavel Volkovitskiy) Date: Wed, 5 Jul 2006 13:11:53 +0400 Subject: [Distutils] setuptools and picard Message-ID: <200607051311.53795.int@mtx.ru> Hello I'm trying to install picard (http://musicbrainz.org/wd/PicardDownload) with python -c "import setuptools;execfile('setup.py')" install --single-version-externally-managed --root=/path/to/root but it breaks with: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: option --single-version-externally-managed not recognized Is that ok and i should use ./setup.py without "import setuptools;execfile('setup.py')" hack, or if this a bug in setuptools that should be fixed? PS: I use setuptools 0.6b3 -- Pavel From pje at telecommunity.com Wed Jul 5 17:16:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Jul 2006 11:16:20 -0400 Subject: [Distutils] setuptools and picard In-Reply-To: <200607051311.53795.int@mtx.ru> Message-ID: <5.1.1.6.0.20060705111243.02080890@sparrow.telecommunity.com> At 01:11 PM 7/5/2006 +0400, Pavel Volkovitskiy wrote: >Hello > >I'm trying to install picard (http://musicbrainz.org/wd/PicardDownload) with >python -c "import setuptools;execfile('setup.py')" install >--single-version-externally-managed --root=/path/to/root >but it breaks with: >usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] > or: -c --help [cmd1 cmd2 ...] > or: -c --help-commands > or: -c cmd --help > >error: option --single-version-externally-managed not recognized > >Is that ok and i should use ./setup.py without "import >setuptools;execfile('setup.py')" hack, or if this a bug in setuptools that >should be fixed? I'm having trouble reproducing the problem, as this tarball: https://helixcommunity.org/download.php/1791/picard-0.6.0.tar.gz doesn't appear to contain a setup.py. Could you give me some more information about what package you are installing from and where its setup.py is coming from? (By the way, --root implies --single-version-externally-managed, so you don't need to use the latter if you are using the former. However, you should only be using --root if you are planning to copy the contents of /path/to/root *to* the actual root, /. Usually this option is used in conjunction with a packaging tool such as RPM. If you aren't planning to copy /path/to/root to /, you shouldn't be using --root either.) From pje at telecommunity.com Wed Jul 5 17:36:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Jul 2006 11:36:03 -0400 Subject: [Distutils] Specification for package indexes? In-Reply-To: References: <5.1.1.6.0.20060623162538.01eabc20@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060705112046.021c5720@sparrow.telecommunity.com> At 04:04 AM 7/5/2006 -0400, Jim Fulton wrote: >On Jun 23, 2006, at 4:51 PM, Jim Fulton wrote: >... >> >>That's a lot of screen scraping. :) >> >>It would be good to capture this as part of the documentation IMO >> >>>I'm considering adding XML-RPC support to easy_install in 0.7, >>>though. PyPI now has a nice XML-RPC API that is more responsive >>>than the web UI, and it supports case-insensitive partial match >>>searches, making it suitable for easy_install to query when a typed- >>>in name doesn't exactly match the spelling of a PyPI entry. >> >>I think that would be much better. > >I just wanted to emphasize that I think this would be a good >idea. Patches welcome. :) Note that there should still be a fallback to the screen scraping code in case of a problem with the XML-RPC, to allow people to continue using static mirrors of PyPI or imitation PyPIs without needing to support XML-RPC. > I was just talking to Richard, and he pointed out that the >current approach is a problem for him, because it means he can't >evolve the pypi UI without risking breaking setuptools. What I would suggest is creating a "microformat" for marking up web pages with sniffable information. For example, adding rel="homepage" and rel="download" to the links that go to those URLs. In other words, invisible hints on the page to supplement the visible information. Then, I could change easy_install to start using the invisible hints, and drop the visible ones, freeing PyPI to evolve the UI again. While the XML-RPC API would be great, I still want easy_install to be able to use a package index that's made from static files, and that requires some kind of screen scraping. So, let's make it invisible scraping of a documented format, so that anybody can use it, with whatever visual formats they like. Currently, easy_install gets most of its information from URLs; the only actual scraping of visible data is of the title, the download MD5's, and the table cells that identify links as being to the home page or download URL (since it needs to specifically identify these in order to spider them). The MD5 information dependency could be removed if PyPI included "#md5=..." at the end of the download URLs; easy_install can see that information and use it. The table cell checking could be removed by adding 'rel="easy_install"' or something like that to the spiderable links. The title checking is used to distinguish pages that list multiple packages from pages that list single packages. I don't have any ready ideas as to how that could or should be represented in a semantic (as opposed to visual) way. Your thoughts? From int at mtx.ru Thu Jul 6 09:37:34 2006 From: int at mtx.ru (Pavel Volkovitskiy) Date: Thu, 6 Jul 2006 11:37:34 +0400 Subject: [Distutils] setuptools and picard In-Reply-To: <5.1.1.6.0.20060705111243.02080890@sparrow.telecommunity.com> References: <5.1.1.6.0.20060705111243.02080890@sparrow.telecommunity.com> Message-ID: <200607061137.34155.int@mtx.ru> On Wednesday 05 July 2006 19:16, you wrote: > >I'm trying to install picard (http://musicbrainz.org/wd/PicardDownload) > > with python -c "import setuptools;execfile('setup.py')" install > >--single-version-externally-managed --root=/path/to/root > >but it breaks with: > >usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] > > or: -c --help [cmd1 cmd2 ...] > > or: -c --help-commands > > or: -c cmd --help > > > >error: option --single-version-externally-managed not recognized > > > >Is that ok and i should use ./setup.py without "import > >setuptools;execfile('setup.py')" hack, or if this a bug in setuptools that > >should be fixed? > > I'm having trouble reproducing the problem, as this tarball: > > https://helixcommunity.org/download.php/1791/picard-0.6.0.tar.gz > > doesn't appear to contain a setup.py. Could you give me some more > information about what package you are installing from and where its > setup.py is coming from? Sorry, i dind't say, i use 0.7.0-beta3 version https://helixcommunity.org/download.php/2009/picard-0.7.0-beta3.tar.gz > > (By the way, --root implies --single-version-externally-managed, so you > don't need to use the latter if you are using the former. However, you > should only be using --root if you are planning to copy the contents of > /path/to/root *to* the actual root, /. Usually this option is used in > conjunction with a packaging tool such as RPM. If you aren't planning to > copy /path/to/root to /, you shouldn't be using --root either.) Yeah, i want to make a package. thanks for tip, i wan't use --single-... for manual tests :) (but i'll use it for packae building as it helps to find errors like this) -- Pavel From richardjones at optusnet.com.au Thu Jul 6 11:09:23 2006 From: richardjones at optusnet.com.au (richardjones at optusnet.com.au) Date: Thu, 06 Jul 2006 19:09:23 +1000 Subject: [Distutils] [Catalog-sig] Specification for package indexes? Message-ID: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://mail.python.org/pipermail/distutils-sig/attachments/20060706/b5b6bc57/attachment.pot From ronaldoussoren at mac.com Thu Jul 6 13:39:16 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 6 Jul 2006 13:39:16 +0200 Subject: [Distutils] md5 checksums through PyPI XML-RPC interface Message-ID: <1F6714AF-55F7-4D78-AA54-4685A96B8DCA@mac.com> Hi, I'm building a tool that interfaces with PyPI and noticed that while it is possible to fetch a lot of information through XML-RPC, the MD5 checksum of files isn't amongst that. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2157 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060706/c394c1f0/attachment.bin From pje at telecommunity.com Thu Jul 6 15:47:34 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 09:47:34 -0400 Subject: [Distutils] setuptools and picard In-Reply-To: <200607061137.34155.int@mtx.ru> References: <5.1.1.6.0.20060705111243.02080890@sparrow.telecommunity.com> <5.1.1.6.0.20060705111243.02080890@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060706094054.01f0ce98@sparrow.telecommunity.com> At 11:37 AM 7/6/2006 +0400, Pavel Volkovitskiy wrote: >On Wednesday 05 July 2006 19:16, you wrote: > > > >I'm trying to install picard (http://musicbrainz.org/wd/PicardDownload) > > > with python -c "import setuptools;execfile('setup.py')" install > > >--single-version-externally-managed --root=/path/to/root > > >but it breaks with: > > >usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] > > > or: -c --help [cmd1 cmd2 ...] > > > or: -c --help-commands > > > or: -c cmd --help > > > > > >error: option --single-version-externally-managed not recognized > > > > > >Is that ok and i should use ./setup.py without "import > > >setuptools;execfile('setup.py')" hack, or if this a bug in setuptools that > > >should be fixed? > > > > I'm having trouble reproducing the problem, as this tarball: > > > > https://helixcommunity.org/download.php/1791/picard-0.6.0.tar.gz > > > > doesn't appear to contain a setup.py. Could you give me some more > > information about what package you are installing from and where its > > setup.py is coming from? > >Sorry, i dind't say, i use 0.7.0-beta3 version >https://helixcommunity.org/download.php/2009/picard-0.7.0-beta3.tar.gz Ah. This setup.py uses a custom "install" command, and setuptools doesn't override that. So the reason you're getting the error message is that picard's install command doesn't support the --single-version-externally-managed option. The only way to work around this and force picard to use setuptools would be to monkeypatch, e.g. something like: python -c "import setuptools.command.install, distutils.command.install; distutils.command.install.install = setuptools.command.install.install; execfile('setup.py')" install --root=/path/to/root This would force picard to subclass setuptools' install command instead of distutils'. However, even this change may not be sufficient, as it appears that picard's setup script wants to install data to a /usr/share/locale or something similar. I don't think it likely that you'll get this to work with setuptools, which is designed to package libraries, not applications. But that's probably okay, because I don't really see why you need to wrap this with setuptools. Since you're building a system package anyway, it should suffice to just use a regular --root install without setuptools. > > > > (By the way, --root implies --single-version-externally-managed, so you > > don't need to use the latter if you are using the former. However, you > > should only be using --root if you are planning to copy the contents of > > /path/to/root *to* the actual root, /. Usually this option is used in > > conjunction with a packaging tool such as RPM. If you aren't planning to > > copy /path/to/root to /, you shouldn't be using --root either.) > >Yeah, i want to make a package. >thanks for tip, i wan't use --single-... for manual tests :) >(but i'll use it for packae building as it helps to find errors like this) From fdrake at gmail.com Thu Jul 6 15:58:23 2006 From: fdrake at gmail.com (Fred Drake) Date: Thu, 6 Jul 2006 09:58:23 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au> References: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au> Message-ID: <9cee7ab80607060658i31dfd34l4c1eeed7e5330129@mail.gmail.com> On 7/6/06, richardjones at optusnet.com.au wrote: > Phillip J. Eby wrote: > > Patches welcome. :) Note that there should still be a fallback to the > > screen scraping code in case of a problem with the XML-RPC, to allow > > people > > to continue using static mirrors of PyPI or imitation PyPIs without > > needing > > to support XML-RPC. > > Why? So we can easily have alternate or additional package repositories implemented simply as (simple) HTML files and the downloadable packages. We want to be able to have an internal repository that plays the easy_install game without running our own PyPI. -Fred -- Fred L. Drake, Jr. "Every sin is the result of a collaboration." --Lucius Annaeus Seneca From pje at telecommunity.com Thu Jul 6 16:03:56 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 10:03:56 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au> Message-ID: <5.1.1.6.0.20060706094755.02bf1d48@sparrow.telecommunity.com> At 07:09 PM 7/6/2006 +1000, richardjones at optusnet.com.au wrote: >Phillip J. Eby wrote: > > Patches welcome. :) Note that there should still be a fallback to the > > screen scraping code in case of a problem with the XML-RPC, to allow > > people > > to continue using static mirrors of PyPI or imitation PyPIs without > > needing > > to support XML-RPC. > >Why? Why not? ;) From easy_install's point of view, PyPI is just a place to find links for a given package name. Preferably links that go directly to downloads, but also to pages that might contain downloads. If someone doesn't want to use PyPI as the source of download links, shouldn't they be able to use their own, without having to implement an XML-RPC interface? Actually, the question of "how do I get easy_install to use something other than PyPI?" has been becoming somewhat of a FAQ recently. Well, two people have asked about it in the last couple of weeks, anyway. And it would've sucked to have to say "well, first you need an XML-RPC server..." :) Nonetheless, there are various aspects of easy_install's behavior and performance that could be significantly improved by using XML-RPC, so I definitely want it to do that in 0.7. I'm just wary of removing the existing behavior until it's clear that it's unnecessary for it to. From richardjones at optusnet.com.au Thu Jul 6 17:03:33 2006 From: richardjones at optusnet.com.au (richardjones at optusnet.com.au) Date: Fri, 07 Jul 2006 01:03:33 +1000 Subject: [Distutils] [Catalog-sig] Specification for package indexes? Message-ID: <200607061503.k66F3X7S005870@mail20.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://mail.python.org/pipermail/distutils-sig/attachments/20060707/d96eaade/attachment.asc From pje at telecommunity.com Thu Jul 6 18:56:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 12:56:03 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <200607061503.k66F3X7S005870@mail20.syd.optusnet.com.au> Message-ID: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> At 01:03 AM 7/7/2006 +1000, richardjones at optusnet.com.au wrote: > > Phillip J. Eby wrote: > > Why not? ;) > >That was actually what I was afraid the reasoning was ;) > >I guess I just go all wobbly in the knees at the thought of having to >maintain a "screen scraping" interface. You don't need to -- at least not in the long term. Once setuptools 0.7 supports the XML-RPC interface, it won't need the other scraping tricks to read PyPI. Those would be left in for people who are creating their own package indexes, not constraining further development of PyPI itself. Please keep in mind that easy_install makes *extremely* minimal assumptions about PyPI's interface: 1. It assumes that baseURL/projectname will get to the current version of projectname, or a page with a list of projectname's active versions 2. It assumes that links within PyPI of the form baseURL/something1/something2 are links to version 'something2' of a project named 'something1' 3. It assumes that going to baseURL directly will result in a page with links to all available packages in the form described in #2. 4. It assumes that if baseURL/projectname returns a page containing the text "Index of Packages", it is a list of links of the form described in #2. 5. It looks for and follows the first links following the strings "Home Page" and "Download URL" in a project page. 6. It makes assumptions about how to find MD5 data on a PyPI page, but if it fails to do so, it simply won't check the MD5 of downloads. Also note that even with an XML-RPC interface, easy_install will *still* need to read an HTML page to gather links, because it's valid for people to provide links in their long_description using reStructuredText. It's just that assumptions 1, 3, and 4 (and maybe 5) would not be necessary. Also note that in a pinch, you can put the strings easy_install is looking for inside HTML comments. Easy_install really isn't that bright. ;) However, if you can provide *all* of this data via the API (including an html-formatted long description), then the screen scraping can go away as far as PyPI is concerned. >Funnily enough, Johannes Gisjbers, Andrew Dalke and I were talking about >this very issue last night. I proposed that we detect the user-agent of >the setuptools client, and in response send back really minimalist HTML >(no surrounding page template). Probably overkill, but this may have been >after we'd had beer :) There's a simpler solution that could be implemented: adding a 'rel="easy-install"' attribute to links that easy_install should follow. Currently, those links are the project's home page URL, download URL, and the links to specific versions that show up when you go to a project that has multiple active versions. Adding it to these, and *only* these links would give easy_install enough information to do the right thing. However, support would have to wait for setuptools 0.7 anyhow, so there's little reason to do this. Hm. I just tried to make multiple versions of PEAK active, and it seems like you can't get the page that lists multiple versions any more. No wonder some people have been having problems downloading older versions of certain packages. :( How are people supposed to get to older package versions now? That is, what's the point of being able to have multiple active versions if you can't find them? Is this an intended change, or a bug? >Could you provide a clear list of all the specific changes you wish for us >to make at the Sprint? I've provided a list above of what changes I want you *not* to make. How's that? ;) > > Nonetheless, there are various aspects of easy_install's behavior and > > performance that could be significantly improved by using XML-RPC, so I > > definitely want it to do that in 0.7. I'm just wary of removing the > > existing behavior until it's clear that it's unnecessary for it to. > >Oh - another thing that occurred to me -- does setuptools auto update itself? What do you mean? You can run "easy_install -u setuptools" to upgrade to the latest release at any time. But it doesn't go out looking for updates on its own. From fccoelho at gmail.com Thu Jul 6 19:09:12 2006 From: fccoelho at gmail.com (Flavio Codeco Coelho) Date: Thu, 06 Jul 2006 14:09:12 -0300 Subject: [Distutils] f2py and setuptools Message-ID: <44AD43B8.8050303@fiocruz.br> Hi, is there a way to handle the compilation of f2py extension modules from setup tools? Or do I have to chose between numpy.distutils.core and setuptools? thanks Fl?vio From cookedm+news at physics.mcmaster.ca Thu Jul 6 19:47:41 2006 From: cookedm+news at physics.mcmaster.ca (David M. Cooke) Date: Thu, 06 Jul 2006 13:47:41 -0400 Subject: [Distutils] f2py and setuptools References: <44AD43B8.8050303@fiocruz.br> Message-ID: Flavio Codeco Coelho writes: > Hi, > > is there a way to handle the compilation of f2py extension modules from > setup tools? > > Or do I have to chose between numpy.distutils.core and setuptools? numpy.distutils interoperates (mostly) ok with setuptools. Just do 'import setuptools' before 'import numpy.distutils'. Although, if you run into problems, please report them to the numpy list (numpy-discussion at lists.sourceforge.net) and we'll see what we can do. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke |cookedm(at)physics(dot)mcmaster(dot)ca From fccoelho at gmail.com Thu Jul 6 20:53:39 2006 From: fccoelho at gmail.com (Flavio Codeco Coelho) Date: Thu, 06 Jul 2006 15:53:39 -0300 Subject: [Distutils] f2py and setuptools In-Reply-To: References: <44AD43B8.8050303@fiocruz.br> Message-ID: <44AD5C33.9060003@fiocruz.br> David M. Cooke wrote: > Flavio Codeco Coelho writes: > > > > numpy.distutils interoperates (mostly) ok with setuptools. Just do > 'import setuptools' before 'import numpy.distutils'. > > Although, if you run into problems, please report them to the numpy > list (numpy-discussion at lists.sourceforge.net) and we'll see what we > can do. > > Actually I would have to do this: from setuptoos import setup from numpy.distutils.core import setup, Extension Will this work? i.e. is the setup class from setuptools equivalent to the setup class from Numpy? thanks, Fl?vio -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060706/2f0cce38/attachment.html From cookedm at physics.mcmaster.ca Thu Jul 6 20:56:57 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 6 Jul 2006 14:56:57 -0400 Subject: [Distutils] f2py and setuptools In-Reply-To: <44AD5C33.9060003@fiocruz.br> References: <44AD43B8.8050303@fiocruz.br> <44AD5C33.9060003@fiocruz.br> Message-ID: <20060706145657.0ae9101b@arbutus.physics.mcmaster.ca> On Thu, 06 Jul 2006 15:53:39 -0300 Flavio Codeco Coelho wrote: > David M. Cooke wrote: > > Flavio Codeco Coelho writes: > > > > > > > > numpy.distutils interoperates (mostly) ok with setuptools. Just do > > 'import setuptools' before 'import numpy.distutils'. > > > > Although, if you run into problems, please report them to the numpy > > list (numpy-discussion at lists.sourceforge.net) and we'll see what we > > can do. > > > > > Actually I would have to do this: > > from setuptoos import setup > from numpy.distutils.core import setup, Extension > > Will this work? i.e. is the setup class from setuptools equivalent to > the setup class from Numpy? Yes; if numpy finds that setuptools has already been imported [1], you can use the numpy.distutils.setup and numpy.distutils.Extension. You don't need to import setup from setuptools; this should be enough: import setuptools from numpy.distutils.core import setup, Extension [1] this is the behaviour in the current svn version. Previous versions would try to import setuptools, and use it if it succeeded. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From richardjones at optusnet.com.au Fri Jul 7 11:41:50 2006 From: richardjones at optusnet.com.au (richardjones at optusnet.com.au) Date: Fri, 07 Jul 2006 19:41:50 +1000 Subject: [Distutils] [Catalog-sig] Specification for package indexes? Message-ID: <200607070941.k679fo2f025860@mail23.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://mail.python.org/pipermail/distutils-sig/attachments/20060707/3fa8eff3/attachment.asc From jim at zope.com Fri Jul 7 12:55:44 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 7 Jul 2006 06:55:44 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> Message-ID: <1DEFC153-D9BB-4FA9-A1FA-65E5F4056AF2@zope.com> I'd like to suggest that we take a step back. It feels as though we are reacting rather than designing. I think we have the following goals: 1. setuptools should be able to read indexes robustly and efficiently. 2. It should be straightforward, and preferably *easy* for people to implement their own indexes. This is very important to me. :) Perhaps: 3. It should be easy to mirror an index 4. It should be possible to create a read index as a static HTTP server. And I suggest: 5. It should be possible to provide an end-user experience for an index without affecting the setuptools interface 4. It should be possible to write other setuptools-like applications for accessing indexes. This means that the web-service (small w-s) should be well defined and/or that setuptools should expose a Python API for accessing indexes. From a design perspective: a. screen scraping is bad b. the web API should be simple and well defined. I suggest, as others have suggested, that we create an *alternate* web API for reading an index focussed on cleanliness and on making the API as easy as possible to implement for both index and client developers. If we agree with all of the goals stated above, I think this should be static HTTP interface using XHTML or some other XML dialect. Perhaps we could even use specific HTML class attrs to make it possible to combine the pypi and user interfaces if an index implementor desires. Thoughts? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jul 7 17:52:29 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 11:52:29 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <200607070941.k679fo2f025860@mail23.syd.optusnet.com.au> Message-ID: <5.1.1.6.0.20060707114135.02d91280@sparrow.telecommunity.com> At 07:41 PM 7/7/2006 +1000, richardjones at optusnet.com.au wrote: > > 3. It assumes that going to baseURL directly will result in a page with > > links to all available packages in the form described in #2. > >This has been removed as it seems completely unnecessary (a flat listing >of all 1400+ packages, that is). The XML-RPC interface provides the >functionality you require here. Note that this will BREAK easy_install in the field, as it will no longer be possible for easy_install to find packages with odd punctuation in their names, or which the user has incorrectly specified the case of. E.g. if someone asks for "sqlobject" instead of "SQLObject", it will no longer work. This is a pretty serious breakage. If you really *must* remove this, then you need to add name canonicalization so that going to /pypi/sqlobject works the same as /pypi/SQLObject. Similarly, going to "/pypi/foo-bar" should work the same as "/pypi/Foo & Bar", and so on. That is, a case-insensitive, "safe_name()" match (see pkg_resources code or docs for the definition of safe_name()). If you can't support this (which I've previously asked you to over the last year so that I could remove the list dependency), PLEASE put the package list back, because you just broke easy_install's ability to support user friendly names. You've known for a *year* that easy_install depended on this feature of PyPI: http://mail.python.org/pipermail/catalog-sig/2005-June/000654.html If you needed it to go away, giving me some notice would have been nice. Not just to me, but to all the people who use PyPI via easy_install. This gratuitous breakage is not nice. From pje at telecommunity.com Fri Jul 7 18:07:19 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 12:07:19 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <5.1.1.6.0.20060707114135.02d91280@sparrow.telecommunity.co m> References: <200607070941.k679fo2f025860@mail23.syd.optusnet.com.au> Message-ID: <5.1.1.6.0.20060707115709.03fe2ea8@sparrow.telecommunity.com> At 11:52 AM 7/7/2006 -0400, Phillip J. Eby wrote: >If you needed it to go away, giving me some notice would have been >nice. Not just to me, but to all the people who use PyPI via >easy_install. This gratuitous breakage is not nice. It looks like easy_install is still actually working at the moment; I guess that means you kept the list working if the user-agent is easy_install. Thank you, and sorry for jumping to conclusions. I took "This has been removed" to mean that it had been, well, removed. :) I will mention again, though, that it *can* be removed entirely, without needing any in-the-field upgrades, if project name matches in URLs can be made case-insensitive and canonical via safe_name: def safe_name(name): """Convert an arbitrary string to a standard distribution name Any runs of non-alphanumeric/. characters are replaced with a single '-'. """ return re.sub('[^A-Za-z0-9.]+', '-', name) If you can fall back to matching safe_name(pkg_name).lower() against search_name.lower() if the URL is not exact, then easy_install will not need the full package listing *or* the XML-RPC interface. It will work as-is in the field today, with no patching required. The only thing the full package listing is used for is to do this search. (Likewise, it would've been the main thing the XML-RPC API would've been used for.) From pje at telecommunity.com Fri Jul 7 18:18:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 12:18:32 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <1DEFC153-D9BB-4FA9-A1FA-65E5F4056AF2@zope.com> References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> At 06:55 AM 7/7/2006 -0400, Jim Fulton wrote: > From a design perspective: > >a. screen scraping is bad As long as you define "screen scraping" as "dependency on visible characteristics of HTML", then I agree. easy_install shouldn't be relying on the visible bits of HTML that it currently uses to scope out PyPI. Relying on a particular URL layout is not screen-scraping, though, and using the URL layout as part of the API has some good properties for ease of implementation in static form. So does using href's to obtain link information. What we should be doing is adding non-visible markup (e.g. class="" or rel="") information to the links to allow index creators to direct easy_install without affecting visible page characteristics. >b. the web API should be simple and well defined. > >I suggest, as others have suggested, that we create an *alternate* >web API for reading an index focussed on cleanliness and on making >the API as easy as possible to implement for both index and client >developers. If we agree with all of the goals stated above, I think >this should be static HTTP interface using XHTML or some other XML >dialect. Perhaps we could even use specific HTML class attrs to >make it possible to combine the pypi and user interfaces if an index >implementor desires. > >Thoughts? +1 on static pages. I don't, however, see a reason to require valid XML. Or rather, I don't expect to implement XML parsing in easy_install; if the spec is too complex to implement with regular expression matching, it's probably too complex for people to throw together an index with what's at hand. In particular, I'd like it to be practical to put together a simple index just using Apache's built-in directory indexes, as long as they use the right URL hierarchy. That means that class or rel attributes should only be required for links that are requesting non-index pages to be spidered. From richardjones at optusnet.com.au Fri Jul 7 18:51:27 2006 From: richardjones at optusnet.com.au (richardjones at optusnet.com.au) Date: Sat, 08 Jul 2006 02:51:27 +1000 Subject: [Distutils] [Catalog-sig] Specification for package indexes? Message-ID: <200607071651.k67GpRvT020914@mail20.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://mail.python.org/pipermail/distutils-sig/attachments/20060708/a445612a/attachment.asc From richardjones at optusnet.com.au Fri Jul 7 19:01:37 2006 From: richardjones at optusnet.com.au (richardjones at optusnet.com.au) Date: Sat, 08 Jul 2006 03:01:37 +1000 Subject: [Distutils] [Catalog-sig] Specification for package indexes? Message-ID: <200607071701.k67H1b22026443@mail16.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://mail.python.org/pipermail/distutils-sig/attachments/20060708/c514dd7d/attachment.pot From jim at zope.com Fri Jul 7 19:32:42 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 7 Jul 2006 13:32:42 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> Message-ID: On Jul 7, 2006, at 12:18 PM, Phillip J. Eby wrote: > At 06:55 AM 7/7/2006 -0400, Jim Fulton wrote: >> From a design perspective: >> >> a. screen scraping is bad > > As long as you define "screen scraping" as "dependency on visible > characteristics of HTML", then I agree. easy_install shouldn't be > relying on the visible bits of HTML that it currently uses to scope > out PyPI. Yup > Relying on a particular URL layout is not screen-scraping, though, > and using the URL layout as part of the API has some good > properties for ease of implementation in static form. So does > using href's to obtain link information. Yes. > What we should be doing is adding non-visible markup (e.g. class="" > or rel="") information to the links to allow index creators to > direct easy_install without affecting visible page characteristics. Yes >> b. the web API should be simple and well defined. >> >> I suggest, as others have suggested, that we create an *alternate* >> web API for reading an index focussed on cleanliness and on making >> the API as easy as possible to implement for both index and client >> developers. If we agree with all of the goals stated above, I think >> this should be static HTTP interface using XHTML or some other XML >> dialect. Perhaps we could even use specific HTML class attrs to >> make it possible to combine the pypi and user interfaces if an index >> implementor desires. >> >> Thoughts? > > +1 on static pages. I don't, however, see a reason to require > valid XML. Or rather, I don't expect to implement XML parsing in > easy_install; if the spec is too complex to implement with regular > expression matching, it's probably too complex for people to throw > together an index with what's at hand. In particular, I'd like it > to be practical to put together a simple index just using Apache's > built-in directory indexes, as long as they use the right URL > hierarchy. That means that class or rel attributes should only be > required for links that are requesting non-index pages to be spidered. I would find parsing much easier with an XML parser than with regular expressions. I think it would be much more robust too. I do want to see something that is well documented and pretty easy to implement. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jul 7 20:02:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 14:02:43 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <200607071701.k67H1b22026443@mail16.syd.optusnet.com.au> Message-ID: <5.1.1.6.0.20060707134531.02d1d778@sparrow.telecommunity.com> At 03:01 AM 7/8/2006 +1000, richardjones at optusnet.com.au wrote: >And I will mention again that I'm not willing to just whack this code into >the package index without some thought to to ramifications given that >distutils has no such name mangling or limitation. ...which is arguably a serious bug in the distutils. Actually, it's several bugs having to do with inconsistent filename mangling throughout the distutils' various commands, and a design limitation of not anticipating that people would register Python packages with name's like "Bob's Incredible Package for Python". :) The ramifications of adding a fallback search to URL lookup (and you need *only* add it to URL lookup for "/pypi/projectname") are these: 1. Exact matches will work exactly as they do now 2. URLs that now produce "Not Found" would produce a list of links that easy_install is already capable of sorting through -- just a shorter one than the whole package index 3. There is no ramification three. :) 4. Users who manually type a URL to find a package will get a nice helpful list of links. :) 5. You can remove the full package listing. >We will have potential (however remote) of name collision and we'll have >to deal with that somehow. Yes, and we should deal with it by rejecting registration of colliding names. The difference between "you can't register an identically-named package" and "you can't register a package that differs only in letter case and punctuation from another package" is very small, but also very *good* for human users, who are not going to be able to remember whether they want "Bob's very Incredible Package" or "Bob's very incredible package", or be able to tell the difference between "My Super Package" and "My Super Package" at a glance. (One of those names has an extra space in it, in case you can't tell.) In truth, the name collision problem is already *possible*, and PyPI should simply reject registration for colliding package names. Allowing people to register say, "SQLObject" and "sqlobject" or "Zope" and "zope" or "foo bar" and "Foo!Bar" as if they were actually different packages isn't doing anybody a favor. Meanwhile, if you go to /pypi/SQLObject now, you get back a list of links that includes SQLObject2, so there's obviously some kind of mangling taking place for URL searches already. If you're concerned about it being ambiguous, just make the fallback search result in a list of matching packages, *exactly the same way that going to "SQLObject" does now*, just with a broader match criterion: http://cheeseshop.python.org/pypi/SQLObject From fdrake at gmail.com Fri Jul 7 20:04:12 2006 From: fdrake at gmail.com (Fred Drake) Date: Fri, 7 Jul 2006 14:04:12 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> Message-ID: <9cee7ab80607071104wb987d3avd2dd9e872822346c@mail.gmail.com> On 7/7/06, Jim Fulton wrote: > > +1 on static pages. I don't, however, see a reason to require > > valid XML. Or rather, I don't expect to implement XML parsing in > > easy_install; if the spec is too complex to implement with regular > > expression matching, it's probably too complex for people to throw > > together an index with what's at hand. In particular, I'd like it > > to be practical to put together a simple index just using Apache's > > built-in directory indexes, as long as they use the right URL > > hierarchy. That means that class or rel attributes should only be > > required for links that are requesting non-index pages to be spidered. > > I would find parsing much easier with an XML parser than with > regular expressions. > I think it would be much more robust too. XHTML would be best, though I agree we shouldn't care about validity so much as just well-formedness (which is required). I think it should be possible to do it with valid XHTML, though, since whether that's desired or not is a python.org policy concern. (Not that I suspect we'll ever really care about that.) Of course, it should be possible to parse with htmllib and HTMLParser as well. -Fred -- Fred L. Drake, Jr. "Every sin is the result of a collaboration." --Lucius Annaeus Seneca From pje at telecommunity.com Fri Jul 7 20:31:01 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 14:31:01 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <9cee7ab80607071104wb987d3avd2dd9e872822346c@mail.gmail.com > References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> At 02:04 PM 7/7/2006 -0400, Fred Drake wrote: >On 7/7/06, Jim Fulton wrote: > > > +1 on static pages. I don't, however, see a reason to require > > > valid XML. Or rather, I don't expect to implement XML parsing in > > > easy_install; if the spec is too complex to implement with regular > > > expression matching, it's probably too complex for people to throw > > > together an index with what's at hand. In particular, I'd like it > > > to be practical to put together a simple index just using Apache's > > > built-in directory indexes, as long as they use the right URL > > > hierarchy. That means that class or rel attributes should only be > > > required for links that are requesting non-index pages to be spidered. > > > > I would find parsing much easier with an XML parser than with > > regular expressions. > > I think it would be much more robust too. > >XHTML would be best, though I agree we shouldn't care about validity >so much as just well-formedness (which is required). I think it >should be possible to do it with valid XHTML, though, since whether >that's desired or not is a python.org policy concern. (Not that I >suspect we'll ever really care about that.) > >Of course, it should be possible to parse with htmllib and HTMLParser as well. I still think requiring even HTML validity or well-formedness is YAGNI; one could indeed just pull all well-formed URLs from the page. EasyInstall uses this case-insensitive regular expression to find only href'd urls: href\s*=\s*['"]?([^'"> ]+) In the absence of a requirement for more information than this (perhaps coupled with a "rel" attribute in the same element), I'm wary of starting out by requiring even well-formedness, because it's way overkill for the requirements as I understand them. One of the advantage of defining the URL layout as part of the API is that it gives you enough contextual information to decide what links should be followed, and which ones are purely informational. Indeed, the only reason to look at anything *but* hrefs is to indicate that an *external* (i.e. non-index) link should be followed, to spider for other download links. So if following external links is out of scope for the API we want to define, then *any* information other than the URLs in an API page are YAGNI. Now, all of this is based on my assumption that the use case here is somebody wants to throw together a rough-and-ready package index that tools should be able to use to find *downloadable distributions*. If you and Jim have much more elaborate use cases in mind, then of course some well-formedness might be useful. On the other hand, if such rigor is required, then it seems like we should just be using machine-readable data in the first place, rather than using a dual-purpose format like HTML or XHTML. Just go with a specialized XML dialect or some kind of text format (ZConfig? ;) ) and be done with it. From pje at telecommunity.com Fri Jul 7 20:50:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 14:50:36 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.co m> References: <9cee7ab80607071104wb987d3avd2dd9e872822346c@mail.gmail.com > <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060707144633.01edd910@sparrow.telecommunity.com> At 02:31 PM 7/7/2006 -0400, Phillip J. Eby wrote: >On the other hand, if such rigor is required, then it seems like we should >just be using machine-readable data in the first place, rather than using a >dual-purpose format like HTML or XHTML. Just go with a specialized XML >dialect or some kind of text format (ZConfig? ;) ) and be done with it. FWIW, I just discovered DOAP, and it seems like a good basis for this sort of thing. PyPI isn't generating 'file-release' info in its DOAP output, but probably should, if this is going to be used for that sort of thing. Likewise, I think that the reST-to-HTML translation should be carried over into the long-description content. (This is all assuming, of course, that you have use cases for which the richness of DOAP is relevant, anyway, as opposed to just trying to find download links for projects with known names.) From jim at zope.com Fri Jul 7 20:52:51 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 7 Jul 2006 14:52:51 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> Message-ID: <46433695-C678-4CE9-AC62-253D01C3EF69@zope.com> On Jul 7, 2006, at 2:31 PM, Phillip J. Eby wrote: > At 02:04 PM 7/7/2006 -0400, Fred Drake wrote: >> On 7/7/06, Jim Fulton wrote: >> > > +1 on static pages. I don't, however, see a reason to require >> > > valid XML. Or rather, I don't expect to implement XML parsing in >> > > easy_install; if the spec is too complex to implement with >> regular >> > > expression matching, it's probably too complex for people to >> throw >> > > together an index with what's at hand. In particular, I'd >> like it >> > > to be practical to put together a simple index just using >> Apache's >> > > built-in directory indexes, as long as they use the right URL >> > > hierarchy. That means that class or rel attributes should >> only be >> > > required for links that are requesting non-index pages to be >> spidered. >> > >> > I would find parsing much easier with an XML parser than with >> > regular expressions. >> > I think it would be much more robust too. >> >> XHTML would be best, though I agree we shouldn't care about validity >> so much as just well-formedness (which is required). I think it >> should be possible to do it with valid XHTML, though, since whether >> that's desired or not is a python.org policy concern. (Not that I >> suspect we'll ever really care about that.) >> >> Of course, it should be possible to parse with htmllib and >> HTMLParser as well. > > I still think requiring even HTML validity or well-formedness is > YAGNI; one could indeed just pull all well-formed URLs from the > page. EasyInstall uses this case-insensitive regular expression to > find only href'd urls: > > href\s*=\s*['"]?([^'"> ]+) > > In the absence of a requirement for more information than this > (perhaps coupled with a "rel" attribute in the same element), I'm > wary of starting out by requiring even well-formedness, because > it's way overkill for the requirements as I understand them. But I thought we *were* talking about adding rel or class tags so that we could determine information about the intended use of a URL. > One of the advantage of defining the URL layout as part of the API > is that it gives you enough contextual information to decide what > links should be followed, and which ones are purely informational. Perhaps someone should propose an API and we'll see. :) > Indeed, the only reason to look at anything *but* hrefs is to > indicate that an *external* (i.e. non-index) link should be > followed, to spider for other download links. So if following > external links is out of scope for the API we want to define, then > *any* information other than the URLs in an API page are YAGNI. Who said following external links is out of scope. > Now, all of this is based on my assumption that the use case here > is somebody wants to throw together a rough-and-ready package index > that tools should be able to use to find *downloadable > distributions*. If you and Jim have much more elaborate use cases > in mind, then of course some well-formedness might be useful. setuptools has a notion of an index. That notion is not at all well defined. Currently, the index has linkes that are followed to find package links elsewhere. This seems reasonably useful. I dunno. I'm not sure I care. What I do care about is that the index API should be well defined so that we can implement alternate indexes and alternate tools to read indexes. I'm not looking to satisfy use cases beyond what we have now. All I want is an API. :) I'm not bent on XML. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jul 7 22:20:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 16:20:11 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <46433695-C678-4CE9-AC62-253D01C3EF69@zope.com> References: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com> At 02:52 PM 7/7/2006 -0400, Jim Fulton wrote: >On Jul 7, 2006, at 2:31 PM, Phillip J. Eby wrote: > >>At 02:04 PM 7/7/2006 -0400, Fred Drake wrote: >>>On 7/7/06, Jim Fulton wrote: >>> > > +1 on static pages. I don't, however, see a reason to require >>> > > valid XML. Or rather, I don't expect to implement XML parsing in >>> > > easy_install; if the spec is too complex to implement with >>>regular >>> > > expression matching, it's probably too complex for people to >>>throw >>> > > together an index with what's at hand. In particular, I'd >>>like it >>> > > to be practical to put together a simple index just using >>>Apache's >>> > > built-in directory indexes, as long as they use the right URL >>> > > hierarchy. That means that class or rel attributes should >>>only be >>> > > required for links that are requesting non-index pages to be >>>spidered. >>> > >>> > I would find parsing much easier with an XML parser than with >>> > regular expressions. >>> > I think it would be much more robust too. >>> >>>XHTML would be best, though I agree we shouldn't care about validity >>>so much as just well-formedness (which is required). I think it >>>should be possible to do it with valid XHTML, though, since whether >>>that's desired or not is a python.org policy concern. (Not that I >>>suspect we'll ever really care about that.) >>> >>>Of course, it should be possible to parse with htmllib and >>>HTMLParser as well. >> >>I still think requiring even HTML validity or well-formedness is >>YAGNI; one could indeed just pull all well-formed URLs from the >>page. EasyInstall uses this case-insensitive regular expression to >>find only href'd urls: >> >> href\s*=\s*['"]?([^'"> ]+) >> >>In the absence of a requirement for more information than this >>(perhaps coupled with a "rel" attribute in the same element), I'm >>wary of starting out by requiring even well-formedness, because >>it's way overkill for the requirements as I understand them. > >But I thought we *were* talking about adding rel or class tags so >that we >could determine information about the intended use of a URL. Yes -- but they're only needed to support following second-order external links: i.e., links to non-index HTML pages. >>One of the advantage of defining the URL layout as part of the API >>is that it gives you enough contextual information to decide what >>links should be followed, and which ones are purely informational. > >Perhaps someone should propose an API and we'll see. :) I thought I already did. :) Here it is again: baseURL/ should return a page containing href links to projects baseURL/projectname should return a page containing href links to version pages baseURL/projectname/version should return a page with download links (ideally with MD5 info) Links are found via href="" attributes URLs' trailing path components are used to identify distributions. This is a sufficient API to allow querying packages for downloading purposes, as long as all download links are found in the index's pages. Additional information is only needed to allow following external links to *other index pages*. Coincidentally, easy_install is already mostly compatible with such an API; it would mostly be a matter of *removing* things from easy_install, rather than adding them. >>Indeed, the only reason to look at anything *but* hrefs is to >>indicate that an *external* (i.e. non-index) link should be >>followed, to spider for other download links. So if following >>external links is out of scope for the API we want to define, then >>*any* information other than the URLs in an API page are YAGNI. > >Who said following external links is out of scope. Nobody; I was just saying that *if* it were out of scope, the class/rel stuff would become unnecessary. >>Now, all of this is based on my assumption that the use case here >>is somebody wants to throw together a rough-and-ready package index >>that tools should be able to use to find *downloadable >>distributions*. If you and Jim have much more elaborate use cases >>in mind, then of course some well-formedness might be useful. > >setuptools has a notion of an index. That notion is not at all well >defined. It's mostly operationally defined in terms of what PyPI did when it was written. >Currently, the index has linkes that are followed to find package >links elsewhere. >This seems reasonably useful. I dunno. I'm not sure I care. What I >do care >about is that the index API should be well defined so that we can >implement >alternate indexes and alternate tools to read indexes. I'm not >looking to >satisfy use cases beyond what we have now. Sure. I'm just saying we only need something beyond href="" links if they are intended to be followed by tools looking for package links. The reason this is necessary, is that it's not sufficient to just follow links that point outside the package index; PyPI has links on its pages that go to other parts of python.org, so there needs to be something that distinguishes "links that might help find downloads". Links that *are* downloads are detected via URL content. From jim at zope.com Fri Jul 7 22:45:59 2006 From: jim at zope.com (Jim Fulton) Date: Fri, 7 Jul 2006 16:45:59 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com> References: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com> Message-ID: <9A0CD067-84D0-4BF0-A5D9-9306AFCDD42E@zope.com> On Jul 7, 2006, at 4:20 PM, Phillip J. Eby wrote: > At 02:52 PM 7/7/2006 -0400, Jim Fulton wrote: ... >> Perhaps someone should propose an API and we'll see. :) > > I thought I already did. :) Here it is again: > > baseURL/ should return a page containing href links to projects > baseURL/projectname should return a page containing href links to > version pages > baseURL/projectname/version should return a page with download > links (ideally with MD5 info) > Links are found via href="" attributes > URLs' trailing path components are used to identify distributions. Hm. I hadn't seen this before. Perhaps I'm missing some messages from this thread. By "download links", do you mean links to distributions? Or to links to pages containing links to distributions. Can the links to projects, links to version pages, or download links point off site? Can any of these pages contain other links? > This is a sufficient API to allow querying packages for downloading > purposes, as long as all download links are found in the index's > pages. Additional information is only needed to allow following > external links to *other index pages*. so, for example: http://www.python.org/pypi/ZODB3/3.6.0 Has a link to http://www.zope.org/Products/ZODB3.6. Is this a download link? Or an off-site index link. I'm having a little trouble following the jargon. >> setuptools has a notion of an index. That notion is not at all well >> defined. > > It's mostly operationally defined in terms of what PyPI did when it > was written. Right, not well defined. :) I'm not criticizing. What it does was great as a prototype, but it would be good move beyond this. >> Currently, the index has linkes that are followed to find package >> links elsewhere. >> This seems reasonably useful. I dunno. I'm not sure I care. What I >> do care >> about is that the index API should be well defined so that we can >> implement >> alternate indexes and alternate tools to read indexes. I'm not >> looking to >> satisfy use cases beyond what we have now. > > Sure. I'm just saying we only need something beyond href="" links > if they are intended to be followed by tools looking for package > links. > > The reason this is necessary, is that it's not sufficient to just > follow links that point outside the package index; PyPI has links > on its pages that go to other parts of python.org, so there needs > to be something that distinguishes "links that might help find > downloads". Links that *are* downloads are detected via URL content. Right. That's why I think the hrefs we care about should be marked with class attributes or some such. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Jul 8 03:12:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 21:12:03 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <9A0CD067-84D0-4BF0-A5D9-9306AFCDD42E@zope.com> References: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com> <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060707175405.01edc3a8@sparrow.telecommunity.com> At 04:45 PM 7/7/2006 -0400, Jim Fulton wrote: >On Jul 7, 2006, at 4:20 PM, Phillip J. Eby wrote: > >>At 02:52 PM 7/7/2006 -0400, Jim Fulton wrote: >... >>>Perhaps someone should propose an API and we'll see. :) >> >>I thought I already did. :) Here it is again: >> >>baseURL/ should return a page containing href links to projects >>baseURL/projectname should return a page containing href links to >>version pages >>baseURL/projectname/version should return a page with download >>links (ideally with MD5 info) >>Links are found via href="" attributes >>URLs' trailing path components are used to identify distributions. > >Hm. I hadn't seen this before. Perhaps I'm missing some messages from >this thread. > >By "download links", do you mean links to distributions? Yes. >Or to links >to pages containing links to distributions. No, these would be either "index pages", or "external links" >Can the links to projects, links to version pages, or download links >point off site? Download links can be anywhere, since they are identified from the tail of the URL. The links to project or version pages are defined by the URL hierarchy of the API. >Can any of these pages contain other links? All of them can contain download links. Index pages can link to other index pages. Index pages linked to anything else are ignored, unless we allow "external links", in which case a method of identifying them is required. Currently, easy_install identifies only uses two kinds of external links: home page and "download URL". These are identified via HTML snippets that PyPI uses. This is one of only two pieces of "screen scraping" (as opposed to URL inspection and link detection) that easy_install has. (The other is used to distinguish between a page that lists links to projects, from an actual project page, as sometimes PyPI can display the former at a URL that is nominally for the latter.) >>This is a sufficient API to allow querying packages for downloading >>purposes, as long as all download links are found in the index's >>pages. Additional information is only needed to allow following >>external links to *other index pages*. > >so, for example: > > http://www.python.org/pypi/ZODB3/3.6.0 > >Has a link to http://www.zope.org/Products/ZODB3.6. >Is this a download link? Or an off-site index link. I'm having a >little trouble >following the jargon. It's an "external link", and thus only followed if it's seen to be the "home page" or "download URL" on a package version page. >>Sure. I'm just saying we only need something beyond href="" links >>if they are intended to be followed by tools looking for package >>links. >> >>The reason this is necessary, is that it's not sufficient to just >>follow links that point outside the package index; PyPI has links >>on its pages that go to other parts of python.org, so there needs >>to be something that distinguishes "links that might help find >>downloads". Links that *are* downloads are detected via URL content. > >Right. That's why I think the hrefs we care about should be marked >with class >attributes or some such. Yes, as long as we care about supporting the external links. I'm not certain we do, at least for the "third-party index" case. From jim at zope.com Sat Jul 8 13:38:06 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 8 Jul 2006 07:38:06 -0400 Subject: [Distutils] [Catalog-sig] Specification for package indexes? In-Reply-To: <5.1.1.6.0.20060707175405.01edc3a8@sparrow.telecommunity.com> References: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com> <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com> <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com> <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com> <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com> <5.1.1.6.0.20060707175405.01edc3a8@sparrow.telecommunity.com> Message-ID: On Jul 7, 2006, at 9:12 PM, Phillip J. Eby wrote: > At 04:45 PM 7/7/2006 -0400, Jim Fulton wrote: ... >> By "download links", do you mean links to distributions? > > Yes. > > >> Or to links >> to pages containing links to distributions. > > No, these would be either "index pages", or "external links" Which seems to be an important use case now. > >> Can the links to projects, links to version pages, or download links >> point off site? > > Download links can be anywhere, since they are identified from the > tail of the URL. The links to project or version pages are defined > by the URL hierarchy of the API. Hm. Why does it matter? I understand that you want to be able to go to index_url/project first, but I don't see that it matters where versions actually are. For that matter, I could see value in a minimal index that just pointed to external project pages. In which case, going to index_url/project might even be allowed to redirect to an offsite project page. Of course, this couldn't be implemented with a static server, but could still be a valuable option. > >> Can any of these pages contain other links? > > All of them can contain download links. Index pages can link to > other index pages. Index pages linked to anything else are > ignored, unless we allow "external links", in which case a method > of identifying them is required. I think we want external links. We have them now. In fact, I think there is value in a project index that has no distributions or even version information but provides a central place to find project pages. Note that, in a separate discussion, you pointed out that some considered it bad form to put interim project releases on pypi. If pypi could have links to remote project pages, then those sites could have different policies as needed by a project. > Currently, easy_install identifies only uses two kinds of external > links: home page and "download URL". These are identified via HTML > snippets that PyPI uses. This is one of only two pieces of "screen > scraping" (as opposed to URL inspection and link detection) that > easy_install has. (The other is used to distinguish between a page > that lists links to projects, from an actual project page, as > sometimes PyPI can display the former at a URL that is nominally > for the latter.) > >>> This is a sufficient API to allow querying packages for downloading >>> purposes, as long as all download links are found in the index's >>> pages. Additional information is only needed to allow following >>> external links to *other index pages*. >> >> so, for example: >> >> http://www.python.org/pypi/ZODB3/3.6.0 >> >> Has a link to http://www.zope.org/Products/ZODB3.6. >> Is this a download link? Or an off-site index link. I'm having a >> little trouble >> following the jargon. > > It's an "external link", and thus only followed if it's seen to be > the "home page" or "download URL" on a package version page. Right, which is currently identified by sniffing the surrounding HTML. > >>> Sure. I'm just saying we only need something beyond href="" links >>> if they are intended to be followed by tools looking for package >>> links. >>> >>> The reason this is necessary, is that it's not sufficient to just >>> follow links that point outside the package index; PyPI has links >>> on its pages that go to other parts of python.org, so there needs >>> to be something that distinguishes "links that might help find >>> downloads". Links that *are* downloads are detected via URL >>> content. >> >> Right. That's why I think the hrefs we care about should be marked >> with class >> attributes or some such. > > Yes, as long as we care about supporting the external links. I'm > not certain we do, at least for the "third-party index" case. I think we do. I'm pretty sure we do for pypi and I sure has heck don't want a different api for pypi and for other indexes. I'd really like to see a single index api. I would *like* to see the possibility of allowing off-site (off- index) projects, although I could live without this. I have to say again that all of these details can get quite confusing. Maybe I'm alone in being confused by this, but I don't think so. I've spent a lot of time on and off over the last few months trying to leverage setuptools and now pypi and while I've had a lot of success, it has been harder than I think it should be. I think that this is an impediment to greater adoption of and benefit from setuptools. I think we need to do a good job of documenting and explaining this API. I also think we need to write up some best practices or rational to guide people toward better use of setuptools and pypi together. I'm happy to help with this once we have agreement and once I understand what we agree to. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From mike_mp at zzzcomputing.com Sat Jul 8 20:18:43 2006 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sat, 8 Jul 2006 14:18:43 -0400 Subject: [Distutils] eggs and PYTHONPATH (cont'd from peak list...) Message-ID: Hi - I have some further thoughts on the PYTHONPATH topic Phillip and I were discussing on the PEAK list a few days ago. On Jun 28, 2006, at 4:09 PM, Phillip J. Eby wrote: > Eggs always override non-eggs, because eggs can be controlled in > ways that non-egg paths can't. The order is (roughly): > > eggs on PYTHONPATH > eggs in site-packages > non-eggs on PYTHONPATH > non-eggs in site-packages > I would propose the order be this: eggs on PYTHONPATH non-eggs on PYTHONPATH eggs in site-packages non-eggs in site-packages It is possible that a non-egg install might be named in PYTHONPATH for a package for which newer versions that are setuptools/egg- enabled are installed system-wide. However, I cant think of a possible scenario where a path would explicitly exist in PYTHONPATH, non-egg or egg, where the user would still like the system-wide installation to take precedence, regardless of versioning or anything else. Otherwise why put the path on PYTHONPATH ? (Though Im guessing there is such a scenario, id just like to see what it is). I know that eggs are more powerful than non-egg paths, allowing precedence to take place based on versioning, which is obviously extremely useful; but I think the idea of PYTHONPATH is that system- wide library paths can be explicitly overridden regardless of any versioning logic taking place either within site-packages or elsewhere. For example, if a particular package on a multi-user shell account is upgraded systemwide from its previously non-egg version to a setuptools/egg version, then any users who have applications running on a custom PYTHONPATH setup with dependencies on their locally- installed, non-egg versions will now have a broken install; theyd have to log in, read all the setuptools docs to figure out whats going on, and do the "virtual python" setup in order to fix things. It seems that setuptools redefines the intention of PYTHONPATH from being a "runtime path determination" variable to an "installation path determination" variable; why repurpose PYTHONPATH in this way, especially when many packages are going to exist in egg/non-egg formats? why not use another variable such as SETUPTOOLSPATH ? Im sure theres other reasons im missing for why its the way it is, so I apologize for harping on this. Also was wondering if anyone else had this same question. - mike From pje at telecommunity.com Mon Jul 10 23:50:17 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Jul 2006 17:50:17 -0400 Subject: [Distutils] Mystery solved In-Reply-To: <0857853C-EF89-4786-93E9-E31CBFC5A93D@zope.com> Message-ID: <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> At 08:38 AM 6/24/2006 -0400, Jim Fulton wrote: >Suppose you have some distributions of interest in a directory >somewhere. You ask easy_install to update your packages. You use >find-links to tell it to look in the directory of distributions. By >default, it looks for an index at http://www.python.org/pypi. >Further assume that the distribution you are updating is not in >PyPI. You'll get the following messages: > > Couldn't find index page for 'jimtest' (maybe misspelled?) > Scanning index of all packages (this may take a while) > >And easy install will have made 2 connections, to: > http://www.python.org/pypi/jimtest/, and to http://www.python.org/ pypi/. > >You should be able to easily reproduce this. > >Now, suppose, instead, you provide an index (-i) option telling >easy_install to look somewhere else? Say, http://www.python.org. You >don't get the messages and only a single connection is made.Why? >What's the difference you ask? Well, PyPI (and my test server) >returns a content type of text/plain for 404 results, while most web >servers return text/html. I changed my test server to return HTML >Non-Found results, and the messages went away and I only get a single >connection. Ah, the irony. Your change, and the recent change of PyPI to do the same thing, are actually forcing *erroneous* behavior now, by suppressing the fallback search. :) >So what is the correct behavior? I dunno. The warnings are rather >annoying, so I'm glad I've found out how to make them go away. They'll come back when I release 0.6b4, since it's setuptools' behavior that's broken here. If a response was text/plain, it was doing the right thing, but it wasn't checking text/html responses to see if they were 404 status. >However, I'm pretty sure your intent was to search both / project_name/ >and /. When a server returns an HTML not-found error, >easy_install skips the / check. > >I think there are 2 bugs here: > >1. The scan of the all-package index is skipped if the content-type >of the > if the /project_name/ request is HTML. Yep, that's a bug, and I'm fixing it now. >2. The messages: > > Couldn't find index page for 'jimtest' (maybe misspelled?) > Scanning index of all packages (this may take a while) > > should be info messages, not warning messages. If they > remain warnings, and you fix the first problem, it will be > impossible to avoid the warnings without creating a PyPI > project or creating an index server, and I don't think it was > your intent to require either of these. I don't think a warning > should be issues for correct use of software. Here's the problem. Reducing everything to info messages means there's effectively no control over output detail. I generally use 'warn()' for things that *may* reflect an error in input parameters. So, my take on the above is that although the "Scanning" message could become an info(), the previous one shouldn't. From jim at zope.com Mon Jul 10 23:58:41 2006 From: jim at zope.com (Jim Fulton) Date: Mon, 10 Jul 2006 17:58:41 -0400 Subject: [Distutils] Mystery solved In-Reply-To: <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> References: <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> Message-ID: On Jul 10, 2006, at 5:50 PM, Phillip J. Eby wrote: ... > >> 2. The messages: >> >> Couldn't find index page for 'jimtest' (maybe misspelled?) >> Scanning index of all packages (this may take a while) >> >> should be info messages, not warning messages. If they >> remain warnings, and you fix the first problem, it will be >> impossible to avoid the warnings without creating a PyPI >> project or creating an index server, and I don't think it was >> your intent to require either of these. I don't think a warning >> should be issues for correct use of software. > > Here's the problem. Reducing everything to info messages means > there's effectively no control over output detail. I generally use > 'warn()' for things that *may* reflect an error in input > parameters. So, my take on the above is that although the > "Scanning" message could become an info(), the previous one shouldn't. This means that one always has to use an index. In which case, what is the point of find-links? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Jul 11 00:22:24 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Jul 2006 18:22:24 -0400 Subject: [Distutils] Mystery solved In-Reply-To: References: <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> At 05:58 PM 7/10/2006 -0400, Jim Fulton wrote: >On Jul 10, 2006, at 5:50 PM, Phillip J. Eby wrote: >... >> >>>2. The messages: >>> >>> Couldn't find index page for 'jimtest' (maybe misspelled?) >>> Scanning index of all packages (this may take a while) >>> >>> should be info messages, not warning messages. If they >>> remain warnings, and you fix the first problem, it will be >>> impossible to avoid the warnings without creating a PyPI >>> project or creating an index server, and I don't think it was >>> your intent to require either of these. I don't think a warning >>> should be issues for correct use of software. >> >>Here's the problem. Reducing everything to info messages means >>there's effectively no control over output detail. I generally use >>'warn()' for things that *may* reflect an error in input >>parameters. So, my take on the above is that although the >>"Scanning" message could become an info(), the previous one shouldn't. > >This means that one always has to use an index. In which case, what >is the point of find-links? The point of --find-links is to provide links to unindexed packages. In 0.6b4, you will be able to use 'file:' URLs for a package index, by the way. From pje at telecommunity.com Tue Jul 11 03:09:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Jul 2006 21:09:57 -0400 Subject: [Distutils] "Package Index API" draft Message-ID: <5.1.1.6.0.20060710210032.03ac2e20@sparrow.telecommunity.com> This draft reflects the in-development versions of setuptools 0.7a1 and 0.6b4; it does not describe older setuptools versions except as noted under the "Backward Compatibility" section. The items described under "Backward Compatibility" need to be kept in PyPI until everyone in the field has upgraded to setuptools 0.6b4 or better. (Note that 0.6b4 is not released yet!) Package Index "API" ------------------- Custom package indexes (and PyPI) must follow the following rules for EasyInstall to be able to look up and download packages: 1. Except where stated otherwise, "pages" are HTML or XHTML, and "links" refer to ``href`` attributes. 2. Individual project version pages' URLs must be of the form ``base/projectname/version``, where ``base`` is the package index's base URL. 3. Omitting the ``/version`` part of a project page's URL (but keeping the trailing ``/``) should result in a page that is either: a) The single active version of that project, as though the version had been explicitly included, OR b) A page with links to all of the active version pages for that project. 4. Individual version pages should contain direct links to downloadable distributions where possible. It is explicitly permitted for a project's "long_description" to include URLs, and these should be formatted as HTML links by the package index, as EasyInstall does no special processing to identify what parts of a page are index-specific and which are part of the project's supplied description. 5. Where available, MD5 information should be added to download URLs by appending a fragment identifier of the form ``#md5=...``, where ``...`` is the 32-character hex MD5 digest. EasyInstall will verify that the downloaded file's MD5 digest matches the given value. 6. Individual project version pages should identify any "homepage" or "download" URLs using ``rel="homepage"`` and ``rel="download"`` attributes on the HTML elements linking to those URLs. Use of these attributes will cause EasyInstall to always follow the provided links, unless it can be determined by inspection that they are downloadable distributions. If the links are not to downloadable distributions, they are retrieved, and if they are HTML, they are scanned for download links. They are *not* scanned for additional "homepage" or "download" links, as these are only processed for pages that are part of a package index site. 7. The root URL of the index, if retrieved with a trailing ``/``, must result in a page containing links to *all* projects' active version pages. (Note: This requirement is a workaround for the absence of case- insensitive ``safe_name()`` matching of project names in URL paths. If project names are matched in this fashion (e.g. via the PyPI server, mod_rewrite, or a similar mechanism), then it is not necessary to include this all-packages listing page.) 8. If a package index is accessed via a ``file://`` URL, then EasyInstall will automatically use ``index.html`` files, if present, when trying to read a directory with a trailing ``/`` on the URL. Backward Compatibility ~~~~~~~~~~~~~~~~~~~~~~ Package indexes that wish to support setuptools versions prior to 0.6b4 should also follow these rules: * Homepage and download links must be preceded with ``"Home Page"`` or ``"Download URL"``, in addition to (or instead of) the ``rel=""`` attributes on the actual links. These marker strings do not need to be visible, or uncommented, however! For example, the following is a valid homepage link that will work with any version of setuptools::
  • Home Page: http://sqlobject.org
    • Even though the marker string is in an HTML comment, older versions of EasyInstall will still "see" it and know that the link that follows is the project's home page URL. * The pages described by paragraph 3(b) of the preceding section *must* contain the string ``"Index of Packages"`` somewhere in their text. This can be inside of an HTML comment, if desired, and it can be anywhere in the page. (Note: this string MUST NOT appear on normal project pages, as described in paragraphs 2 and 3(a)!) In addition, for compatibility with PyPI versions that do not use ``#md5=`` fragment IDs, EasyInstall uses the following regular expression to match PyPI's displayed MD5 info (broken onto two lines for readability):: ([^<]+)\n\s+\(md5\) From jim at zope.com Tue Jul 11 09:54:12 2006 From: jim at zope.com (Jim Fulton) Date: Tue, 11 Jul 2006 03:54:12 -0400 Subject: [Distutils] Mystery solved In-Reply-To: <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> References: <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> Message-ID: <626CED7F-2C4E-4C94-86DA-029036BB919A@zope.com> On Jul 10, 2006, at 6:22 PM, Phillip J. Eby wrote: > At 05:58 PM 7/10/2006 -0400, Jim Fulton wrote: > >> On Jul 10, 2006, at 5:50 PM, Phillip J. Eby wrote: >> ... >>> >>>> 2. The messages: >>>> >>>> Couldn't find index page for 'jimtest' (maybe misspelled?) >>>> Scanning index of all packages (this may take a while) >>>> >>>> should be info messages, not warning messages. If they >>>> remain warnings, and you fix the first problem, it will be >>>> impossible to avoid the warnings without creating a PyPI >>>> project or creating an index server, and I don't think it was >>>> your intent to require either of these. I don't think a >>>> warning >>>> should be issues for correct use of software. >>> >>> Here's the problem. Reducing everything to info messages means >>> there's effectively no control over output detail. I generally use >>> 'warn()' for things that *may* reflect an error in input >>> parameters. So, my take on the above is that although the >>> "Scanning" message could become an info(), the previous one >>> shouldn't. >> >> This means that one always has to use an index. In which case, what >> is the point of find-links? > > The point of --find-links is to provide links to unindexed packages. But if you use an unindexed package, you'll get a warning. IMO, you should not get a warning for correct use of software. Users should try to make warnings go away. If you give people warnings that they shouldn't make go away, then they are more likely to ignore warnings that you don't want them to. Either you can't have valid unindexed software, or setuptools shouldn't generate a warning if software isn't in the index. I really find the distinction between indexes and find-links rather puzzling. Personally, I'd like to find a way to merge these two concepts into one by choosing a definition of an index that admits a directory full of distributions. Then we could get rid of the find-links concept and allow 0 or more indexes to be used. This would be much simpler and cleaner IMO. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From lists at hlabs.spb.ru Tue Jul 11 13:24:17 2006 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Tue, 11 Jul 2006 15:24:17 +0400 Subject: [Distutils] Keep downloaded files in case of an error Message-ID: <44B38A61.5050008@hlabs.spb.ru> Hi, Is there a way to keep downloaded files in case of an error? For example in my test setup.py I wrote: ... 'install_requires=["Twisted >= 2.4.0"]' ... Then I did: $ python setup.py develop ... Downloading ... Twisted-2.4.0.tar.bz2 And then few minutes later: Processing Twisted-2.4.0.tar.bz2 error: Not a recognized archive type: .../Twisted-2.4.0.tar.bz2 The downloaded archive disappears. I think it would be useful to keep (or move to some selectable location) downloaded files in case of an error. -- Dmitry Vasiliev (dima at hlabs.spb.ru) http://hlabs.spb.ru From jorge.vargas at gmail.com Tue Jul 11 17:17:43 2006 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Tue, 11 Jul 2006 11:17:43 -0400 Subject: [Distutils] a patch for distutils + gcc + linux, that was never apply, yet will solve many problems. In-Reply-To: <32822fe60606302039n2246fbfay53a3fed9bf50d94f@mail.gmail.com> References: <32822fe60606302039n2246fbfay53a3fed9bf50d94f@mail.gmail.com> Message-ID: <32822fe60607110817x1465d1c8tec85a28890ad31a7@mail.gmail.com> noone cares about this? it's a bug so simple to fix.... On 6/30/06, Jorge Vargas wrote: > > Hello everyone > > I did a package for gentoo (ebuild) a while ago and it has a C/C++ > extension, to compile agains a gecko-compliant mozilla product. > > Long story short our solution (build agains mozilla) seem to had nothing > to do with the real bug which someone posted today. > > It seems there is a bug in distutils itself, because it is passing the > wrong arguments to g++, the most incredible part is that the bug is so > simple to fix yet there is no reply, maybe this got lost in the main python > repository. > > here is the most important part of his post > ---- > > /usr/lib/python2.4/distutils/unixccompiler.py to build. > ** /usr/lib/python2.4/distutils/unixccompiler.py fills the 'CC' executable > > variable by invoking /usr/lib/python2.4/distutils/sysconfig.py > ** /usr/lib/python2.4/distutils/unixccompiler.py contains a condition that > says: > elif compiler[:3] == "gcc" or compiler[:3] == "g++": > > return "-Wl,-R" + dir > ** /usr/lib/python2.4/distutils/sysconfig.py returns the 'CC' executable > variable by reading from /usr/lib/python2.4/config/Makefile > ** If your system is like many others, it will return something like > > 'CC=i686-pc-linux-gnu-gcc -pthread', will never == "gcc" and so the correct > linker options will not be used. > > ---- > if anyone could please verify it and maybe find out why it wasn't apply, > honestly I have very little knowlegde of C and it's integration with python > to be able to verify it. > > his full post is at > http://bugs.gentoo.org/show_bug.cgi?id=131527#c36 > > the original disutil gentoo bug: > http://bugs.gentoo.org/show_bug.cgi?id=138647 > > the patch at sourceforce: > http://sourceforge.net/tracker/index.php?func=detail&aid=1254718&group_id=5470&atid=305470 > > > > thanks. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060711/8c5f0671/attachment.html From pje at telecommunity.com Tue Jul 11 17:25:13 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 11:25:13 -0400 Subject: [Distutils] Mystery solved In-Reply-To: <626CED7F-2C4E-4C94-86DA-029036BB919A@zope.com> References: <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> At 03:54 AM 7/11/2006 -0400, Jim Fulton wrote: >On Jul 10, 2006, at 6:22 PM, Phillip J. Eby wrote: >>>>Here's the problem. Reducing everything to info messages means >>>>there's effectively no control over output detail. I generally use >>>>'warn()' for things that *may* reflect an error in input >>>>parameters. So, my take on the above is that although the >>>>"Scanning" message could become an info(), the previous one >>>>shouldn't. >>> >>>This means that one always has to use an index. In which case, what >>>is the point of find-links? >> >>The point of --find-links is to provide links to unindexed packages. > >But if you use an unindexed package, you'll get a warning. Which means it *may* be an error. See above. If it was an error, it would be an error. The point of a warning is to inform you that something *may* be an error. >IMO, you should not get a warning for correct use of software. Users >should try to make warnings go away. Repeating it doesn't make it so. I'm not convinced that this particular warning (that you may have misspelled the package name because it's not in the index you're using) is in any way harmful. > If you give people warnings that >they shouldn't make go away, Huh? They can put the package in the index, use a different index, not use -U, specify an exact download URL (either directly or via --find-links), etc. There are a huge number of ways *not* to encounter that particular warning. >then they are more likely to ignore >warnings >that you don't want them to. What warnings? There are only about 13 left. :) >Either you can't have valid unindexed software, or setuptools shouldn't >generate a warning if software isn't in the index. This is back to argument by assertion. Please explain to me why this warning is actually bad, rather than simply asserting that it's so. >I really find the distinction between indexes and find-links rather >puzzling. --find-links is used to allow you to point easy_install to a project's non-indexed home page or download page to find links, or to provide other easy_install-processable links, without needing an index. >Personally, I'd like to find a way to merge these two concepts into one >by choosing a definition of an index that admits a directory full of >distributions. Feel free to try to come up with one. However, --find-links allows *multiple* links to be specified, and it is also the basis for the "dependency_links" argument to setup(). --find-links is also a primitive upon which the index facility is built, since index pages are treated more-or-less like --find-links URLs that are automatically generated. At a minimum, merging the concepts would mean allowing multiple index URLs, or else eliminating the idea of an index, and treating all --find-links URLs as though they were the base URL of a package index. If you did that, however, it brings in the question of which of the --find-links URLs should be checked for a /projectname/ subdirectory. All of them? Just the first one that finds a result? None of them, if some other criterion is met? Currently, a remote package search means that all --find-links pages are checked, and the --index-url is searched (by going to "index_url/projectname/"). From ronaldoussoren at mac.com Tue Jul 11 17:42:12 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 11 Jul 2006 17:42:12 +0200 Subject: [Distutils] a patch for distutils + gcc + linux, that was never apply, yet will solve many problems. In-Reply-To: <32822fe60607110817x1465d1c8tec85a28890ad31a7@mail.gmail.com> References: <32822fe60606302039n2246fbfay53a3fed9bf50d94f@mail.gmail.com> <32822fe60607110817x1465d1c8tec85a28890ad31a7@mail.gmail.com> Message-ID: On Jul 11, 2006, at 5:17 PM, Jorge Vargas wrote: > noone cares about this? it's a bug so simple to fix.... There's loads of bugs and patches in the python tracker, and only a limited amount of time that people work on python. The patch you mentioned also sounds like a fix for a very limited audience, which could explain why nobody has seriously looked at it yet. Are there systems other then gentoo that don't use gcc and g++ but some seemingly cross-compiling setup? Ronald > > On 6/30/06, Jorge Vargas wrote: > Hello everyone > > I did a package for gentoo (ebuild) a while ago and it has a C/C++ > extension, to compile agains a gecko-compliant mozilla product. > > Long story short our solution (build agains mozilla) seem to had > nothing to do with the real bug which someone posted today. > > It seems there is a bug in distutils itself, because it is passing > the wrong arguments to g++, the most incredible part is that the > bug is so simple to fix yet there is no reply, maybe this got lost > in the main python repository. > > here is the most important part of his post > ---- > /usr/lib/python2.4/distutils/unixccompiler.py to build. > ** /usr/lib/python2.4/distutils/unixccompiler.py fills the 'CC' > executable > > variable by invoking /usr/lib/python2.4/distutils/sysconfig.py > ** /usr/lib/python2.4/distutils/unixccompiler.py contains a > condition that > says: > elif compiler[:3] == "gcc" or compiler[:3] == "g++": > > return "-Wl,-R" + dir > ** /usr/lib/python2.4/distutils/sysconfig.py returns the 'CC' > executable > variable by reading from /usr/lib/python2.4/config/Makefile > ** If your system is like many others, it will return something like > > 'CC=i686-pc-linux-gnu-gcc -pthread', will never == "gcc" and so the > correct > linker options will not be used.---- > if anyone could please verify it and maybe find out why it wasn't > apply, honestly I have very little knowlegde of C and it's > integration with python to be able to verify it. > > his full post is at > http://bugs.gentoo.org/show_bug.cgi?id=131527#c36 > > the original disutil gentoo bug: > http://bugs.gentoo.org/show_bug.cgi?id=138647 > > the patch at sourceforce: > http://sourceforge.net/tracker/index.php? > func=detail&aid=1254718&group_id=5470&atid=305470 > > > thanks. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From jim at zope.com Tue Jul 11 17:50:04 2006 From: jim at zope.com (Jim Fulton) Date: Tue, 11 Jul 2006 11:50:04 -0400 Subject: [Distutils] Mystery solved In-Reply-To: <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> References: <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> Message-ID: <167B12E2-C2D9-4AAF-ADC5-9B29BA379D1C@zope.com> On Jul 11, 2006, at 11:25 AM, Phillip J. Eby wrote: > At 03:54 AM 7/11/2006 -0400, Jim Fulton wrote: >> On Jul 10, 2006, at 6:22 PM, Phillip J. Eby wrote: >>>>> Here's the problem. Reducing everything to info messages means >>>>> there's effectively no control over output detail. I generally >>>>> use >>>>> 'warn()' for things that *may* reflect an error in input >>>>> parameters. So, my take on the above is that although the >>>>> "Scanning" message could become an info(), the previous one >>>>> shouldn't. >>>> >>>> This means that one always has to use an index. In which case, >>>> what >>>> is the point of find-links? >>> >>> The point of --find-links is to provide links to unindexed packages. >> >> But if you use an unindexed package, you'll get a warning. > > Which means it *may* be an error. See above. If it was an error, > it would be an error. The point of a warning is to inform you that > something *may* be an error. > >> IMO, you should not get a warning for correct use of software. Users >> should try to make warnings go away. > > Repeating it doesn't make it so. I'm not convinced that this > particular warning (that you may have misspelled the package name > because it's not in the index you're using) is in any way harmful. It is definitely so. That is definitely my opinion. :) OK that's an interesting point wrt possible misspellings. If you can find the package via the find links, but not via the index, that seems to me to be a pretty good indication that this is not a misspelling. This is the case I'm worried about. If the package can't be found anywhere, then I agree that a warning is warranted. > >> If you give people warnings that >> they shouldn't make go away, That wasn't clear. If people are using the software correctly, but choosing to find distributions via find-links rather than an index, and they get warnings, then they will always get warnings and tend to ignore them. > > Huh? They can put the package in the index, use a different index, > not use -U, specify an exact download URL (either directly or via -- > find-links), etc. There are a huge number of ways *not* to > encounter that particular warning. I have to use -U to get newer versions of distributions, even if I happen to store distributions in a directory that is not a valid index. In this case, I use find-links and -U together, and I'll get a warning unless I put distributions in an index. >> Either you can't have valid unindexed software, or setuptools >> shouldn't >> generate a warning if software isn't in the index. > > This is back to argument by assertion. Please explain to me why > this warning is actually bad, rather than simply asserting that > it's so. I assert and take as a premise that when users are using software correctly, including not misspelling anything, they should not get a warning. If you can't buy that, then we have an unreconcilable difference. The specific case, which I'll repeat from above, as clearly as I can, is this: - A user chooses not to store their software in an index. - The user places distributions on a web server somewhere. This is just a directory, it is not a valid index. - The user points at their server using find-links - The user has an installation and they want to check for newer versions. - The distributions that they are looking for newer versions of can be found on the server that they name via find-links. In this case, they will get a warning that the distribution they are looking for couldn't be found on the index. They didn't misspell anything, as setuptools should be able to deduce from the fact that their distribution was found on the link server. I don't think that they should get a warning. As far as I'm concerned, this means that distributions must always be stored on index servers and the find-links is just an attractive nuisance. > >> I really find the distinction between indexes and find-links rather >> puzzling. > > --find-links is used to allow you to point easy_install to a > project's non-indexed home page or download page to find links, or > to provide other easy_install-processable links, without needing an > index. But they are unusable without getting warnings whenever you want to check for updates. > >> Personally, I'd like to find a way to merge these two concepts >> into one >> by choosing a definition of an index that admits a directory full of >> distributions. > > Feel free to try to come up with one. However, --find-links allows > *multiple* links to be specified, and it is also the basis for the > "dependency_links" argument to setup(). --find-links is also a > primitive upon which the index facility is built, since index pages > are treated more-or-less like --find-links URLs that are > automatically generated. I don't need to, you already did.... > At a minimum, merging the concepts would mean allowing multiple > index URLs, or else eliminating the idea of an index, Yup. Sound good to me. > and treating all --find-links URLs as though they were the base URL > of a package index. Yes > If you did that, however, it brings in the question of which of > the --find-links URLs should be checked for a /projectname/ > subdirectory. All of them? Just the first one that finds a > result? None of them, if some other criterion is met? I would stop when a result is found. > Currently, a remote package search means that all --find-links > pages are checked, and the --index-url is searched (by going to > "index_url/projectname/"). What is the use case for spreading distributions over multiple servers? Do people really want to do that? I can see providing multiple places to look, because different distributions might be on different servers, but I don't see why distributions for a single project should be spread over multiple servers. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Jul 11 20:07:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 14:07:36 -0400 Subject: [Distutils] Mystery solved In-Reply-To: <167B12E2-C2D9-4AAF-ADC5-9B29BA379D1C@zope.com> References: <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060711133050.01ef0870@sparrow.telecommunity.com> At 11:50 AM 7/11/2006 -0400, Jim Fulton wrote: >OK that's an interesting point wrt possible misspellings. If you can >find the package via the find links, but not via the index, that >seems to me to be a pretty good indication that this is not a >misspelling. This is the case I'm worried about. If the package >can't be found anywhere, then I agree that a warning is warranted. The interesting question there is, should the fallback scan still take place in the absence of the warning? If it *does* take place, then the reason for the scan (and delay) is unexplained. If it does *not* take place, then there is an undesirable change in semantics. Currently, if you have a package called "Bob's Incredible Package", this will be treated by easy_install as being spelled "Bob-s-Incredible-Package", and it will require a top-level index scan to find the right URL. It is also possible to have --find-links pages containing obsolete versions, while PyPI contains the latest version, so removing the scan doesn't seem to be a reasonable option. So, I will simply change the message to an "info" message stating that the index page couldn't be found (rather than a warning suggesting misspelling), *if* easy_install has previously seen at least one valid distribution file or link for the applicable project name. >The specific case, which I'll repeat from above, as clearly as I can, >is this: > >- A user chooses not to store their software in an index. >- The user places distributions on a web server somewhere. This is >just a directory, it is not a valid index. >- The user points at their server using find-links >- The user has an installation and they want to check for newer >versions. >- The distributions that they are looking for newer versions of can >be found on the server that they name via find-links. > >In this case, they will get a warning that the distribution they are >looking for couldn't be found on the index. Okay, this scenario is fixed by changing to an info message as described above. >>>Personally, I'd like to find a way to merge these two concepts >>>into one >>>by choosing a definition of an index that admits a directory full of >>>distributions. >> >>Feel free to try to come up with one. However, --find-links allows >>*multiple* links to be specified, and it is also the basis for the >>"dependency_links" argument to setup(). --find-links is also a >>primitive upon which the index facility is built, since index pages >>are treated more-or-less like --find-links URLs that are >>automatically generated. > >I don't need to, you already did.... No, I presented a straw man to show why it doesn't work. I guess I should've been more explicit in spelling out all the undesirable consequences. >> If you did that, however, it brings in the question of which of >>the --find-links URLs should be checked for a /projectname/ >>subdirectory. All of them? Just the first one that finds a >>result? None of them, if some other criterion is met? > >I would stop when a result is found. Even so, this means O(N x M) web hits, where N is the number of packages and M is the number of --find-links (including dependency links supplied by eggs installed so far). I don't think it's reasonable to hit so many non-existent URLs on non-index servers, and is impolite to the servers' operators. (For example, if they receive a daily report of all 404 errors from their web servers, as I do. This is pretty common on Red Hat boxes using logwatch, for example.) It's particularly unfair since using e.g. http://peak.telecommunity.com/snapshots/ as a --find-links while installing, say TurboGears, would cause a whole host of "index" hits to subdirectories of that URL, even though none of them can or will be found. The fallout from this approach is far worse than any "screen scraping" issues we've had. >What is the use case for spreading distributions over multiple >servers? Do people really want to do that? I can see providing >multiple places to look, because different distributions might be on >different servers, but I don't see why distributions for a single >project should be spread over multiple servers. Platform-specific distributions may be provided by contributors to a project, rather than by the project's author; see, for example, Bob Ippolito's pages for distributing Mac OS X builds of popular Python packages. For this reason, you may have certain pages that you always want included in your --find-links, to be checked in addition to the normal indexes. From pje at telecommunity.com Tue Jul 11 21:12:16 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 15:12:16 -0400 Subject: [Distutils] setuptools 0.6b4 released Message-ID: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> I have just released version 0.6b4 of setuptools, the last beta release of setuptools 0.6. Please upgrade and test at your earliest convenience, as I would like to issue a release candidate version next week. Changes include numerous bug fixes and tweaks for corner cases in easy_install processing, mostly discovered by Jim Fulton in the process of developing his "zc.buildout" tool (a Make-like system that supports simple installation of complex environments). In addition, there is now a formally-documented "package index API" that easy_install supports: http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api So, people who wish to create their own "package indexes" for easy_install can do so, even using static HTML -- even without a web server. Last, but not least, the ability was added to turn off SVN revision numbers or dates from the command line, so that you don't have to edit setup.cfg in order to issue a release. There are a few outstanding requests that I was *not* able to consider for 0.6 because the required changes would've been too disruptive to stability. They have been bumped to the 0.7 list to receive further consideration after 0.6 final is released: * A request to change the PYTHONPATH resolution order * A request to allow installed scripts to use -O * Adding XML-RPC support for PyPI From bob at redivi.com Tue Jul 11 21:33:16 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 11 Jul 2006 12:33:16 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote: > Last, but not least, the ability was added to turn off SVN revision > numbers > or dates from the command line, so that you don't have to edit > setup.cfg in > order to issue a release. Would've been convenient if you said what the option was... took me a few minutes to figure out where to look. Presumably you're referring to the --no-svn-revision option? Options for 'egg_info' command: --egg-base (-e) directory containing .egg-info directories (default: top of the source tree) --tag-svn-revision (-r) Add subversion revision ID to version number --tag-date (-d) Add date stamp (e.g. 20050528) to version number --tag-build (-b) Specify explicit tag to add to version number --no-svn-revision (-R) Don't add subversion revision ID [default] --no-date (-D) Don't include date stamp [default] --tag-build (-b) Specify explicit tag to add to version number Note that tag-build shows up twice, here's a patch: Index: setuptools/command/egg_info.py =================================================================== --- setuptools/command/egg_info.py (revision 50587) +++ setuptools/command/egg_info.py (working copy) @@ -28,7 +28,6 @@ ('no-svn-revision', 'R', "Don't add subversion revision ID [default]"), ('no-date', 'D', "Don't include date stamp [default]"), - ('tag-build=', 'b', "Specify explicit tag to add to version number"), ] boolean_options = ['tag-date', 'tag-svn-revision'] -bob From pje at telecommunity.com Tue Jul 11 21:53:01 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 15:53:01 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060711154900.03a9ff88@sparrow.telecommunity.com> At 12:33 PM 7/11/2006 -0700, Bob Ippolito wrote: >On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote: >>Last, but not least, the ability was added to turn off SVN revision >>numbers >>or dates from the command line, so that you don't have to edit >>setup.cfg in >>order to issue a release. > >Would've been convenient if you said what the option was... took me a >few minutes to figure out where to look. Presumably you're referring >to the --no-svn-revision option? Yep. See also: http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion In particular, the new subsection: http://peak.telecommunity.com/DevCenter/setuptools#making-official-non-snapshot-releases Quick tip: you can use: http://peak.telecommunity.com/DevCenter/setuptools?action=diff to check on the latest changes to setuptools' doc after a release. The wiki also allows you to subscribe to receive change notices via email. This also goes for the EasyInstall and PkgResources doc pages. >Note that tag-build shows up twice, here's a patch: Argh. :( Thanks for catching that. I've fixed it in the trunk and 0.6 branch. From jim at zope.com Tue Jul 11 21:56:48 2006 From: jim at zope.com (Jim Fulton) Date: Tue, 11 Jul 2006 15:56:48 -0400 Subject: [Distutils] Mystery solved In-Reply-To: <5.1.1.6.0.20060711133050.01ef0870@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> <5.1.1.6.0.20060711133050.01ef0870@sparrow.telecommunity.com> Message-ID: On Jul 11, 2006, at 2:07 PM, Phillip J. Eby wrote: > At 11:50 AM 7/11/2006 -0400, Jim Fulton wrote: >> OK that's an interesting point wrt possible misspellings. If you can >> find the package via the find links, but not via the index, that >> seems to me to be a pretty good indication that this is not a >> misspelling. This is the case I'm worried about. If the package >> can't be found anywhere, then I agree that a warning is warranted. > > The interesting question there is, should the fallback scan still > take place in the absence of the warning? If it *does* take place, > then the reason for the scan (and delay) is unexplained. If it > does *not* take place, then there is an undesirable change in > semantics. > > Currently, if you have a package called "Bob's Incredible Package", > this will be treated by easy_install as being spelled "Bob-s- > Incredible-Package", and it will require a top-level index scan to > find the right URL. It is also possible to have --find-links pages > containing obsolete versions, while PyPI contains the latest > version, so removing the scan doesn't seem to be a reasonable option. > > So, I will simply change the message to an "info" message stating > that the index page couldn't be found (rather than a warning > suggesting misspelling), *if* easy_install has previously seen at > least one valid distribution file or link for the applicable > project name. Great! > >> The specific case, which I'll repeat from above, as clearly as I can, >> is this: >> >> - A user chooses not to store their software in an index. >> - The user places distributions on a web server somewhere. This is >> just a directory, it is not a valid index. >> - The user points at their server using find-links >> - The user has an installation and they want to check for newer >> versions. >> - The distributions that they are looking for newer versions of can >> be found on the server that they name via find-links. >> >> In this case, they will get a warning that the distribution they are >> looking for couldn't be found on the index. > > Okay, this scenario is fixed by changing to an info message as > described above. Yup. Cool. >>> If you did that, however, it brings in the question of which of >>> the --find-links URLs should be checked for a /projectname/ >>> subdirectory. All of them? Just the first one that finds a >>> result? None of them, if some other criterion is met? >> >> I would stop when a result is found. > > Even so, this means O(N x M) web hits, where N is the number of > packages and M is the number of --find-links (including dependency > links supplied by eggs installed so far). I don't think it's > reasonable to hit so many non-existent URLs on non-index servers, > and is impolite to the servers' operators. (For example, if they > receive a daily report of all 404 errors from their web servers, as > I do. This is pretty common on Red Hat boxes using logwatch, for > example.) > > It's particularly unfair since using e.g. http:// > peak.telecommunity.com/snapshots/ as a --find-links while > installing, say TurboGears, would cause a whole host of "index" > hits to subdirectories of that URL, even though none of them can or > will be found. > > The fallout from this approach is far worse than any "screen > scraping" issues we've had. Isn't this the approach that's followed now? Aren't all of the find- links searched as well as the index? I suppose you're referring to the search for /projectname, which potentially doubles the number of requests. >> What is the use case for spreading distributions over multiple >> servers? Do people really want to do that? I can see providing >> multiple places to look, because different distributions might be on >> different servers, but I don't see why distributions for a single >> project should be spread over multiple servers. > > Platform-specific distributions may be provided by contributors to > a project, rather than by the project's author; see, for example, > Bob Ippolito's pages for distributing Mac OS X builds of popular > Python packages. For this reason, you may have certain pages that > you always want included in your --find-links, to be checked in > addition to the normal indexes. OK Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From bob at redivi.com Thu Jul 13 06:00:06 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 12 Jul 2006 21:00:06 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: <8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com> On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote: > I have just released version 0.6b4 of setuptools, the last beta > release of > setuptools 0.6. Please upgrade and test at your earliest > convenience, as I > would like to issue a release candidate version next week. It seems there's another bug in 0.6b4. I don't know what conditions cause it, however: File "/Volumes/Data/developer-external/py2app/setuptools-0.6b4- py2.4.egg/setuptools/dist.py", line 274, in fetch_build_egg AttributeError: Distribution instance has no attribute 'dependency_links' It looks like there's no default for dependency_links in some cases.. so that needs to change to a getattr, or it needs to set some default. -bob From bob at redivi.com Thu Jul 13 06:09:20 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 12 Jul 2006 21:09:20 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com> References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com> Message-ID: <1C91F591-0072-4E7C-9E5B-F1DC0F79FA3D@redivi.com> On Jul 12, 2006, at 9:00 PM, Bob Ippolito wrote: > > On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote: > >> I have just released version 0.6b4 of setuptools, the last beta >> release of >> setuptools 0.6. Please upgrade and test at your earliest >> convenience, as I >> would like to issue a release candidate version next week. > > It seems there's another bug in 0.6b4. I don't know what conditions > cause it, however: > > File "/Volumes/Data/developer-external/py2app/setuptools-0.6b4- > py2.4.egg/setuptools/dist.py", line 274, in fetch_build_egg > AttributeError: Distribution instance has no attribute > 'dependency_links' > > It looks like there's no default for dependency_links in some cases.. > so that needs to change to a getattr, or it needs to set some default. Ok, it seems this happens whenever you do develop, install, etc. when all dependencies aren't available. easy_install seems to work though. This patch seems to make everything work as expected: =================================================================== --- setuptools/dist.py (revision 50587) +++ setuptools/dist.py (working copy) @@ -271,7 +271,7 @@ for key in opts.keys(): if key not in keep: del opts[key] # don't use any other settings - if self.dependency_links: + if getattr(self, 'dependency_links', None): links = self.dependency_links[:] if 'find_links' in opts: links = opts['find_links'][1].split() + links -bob From bob at redivi.com Thu Jul 13 09:43:32 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 00:43:32 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: <2E9807AF-1C61-4E49-88A9-DC32CD9D0336@redivi.com> On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote: > I have just released version 0.6b4 of setuptools, the last beta > release of > setuptools 0.6. Please upgrade and test at your earliest > convenience, as I > would like to issue a release candidate version next week. Here's another patch I'd like to see ASAP. Some vendors have Q&A procedures that disallow empty files without special exception, so we should ensure that setuptools doesn't create empty safety flag files. Index: setuptools/command/bdist_egg.py =================================================================== --- setuptools/command/bdist_egg.py (revision 50611) +++ setuptools/command/bdist_egg.py (working copy) @@ -360,7 +360,9 @@ if safe is None or bool(safe)<>flag: os.unlink(fn) elif safe is not None and bool(safe)==flag: - open(fn,'w').close() + f = open(fn,'w') + f.write(fn) + f.close() safety_flags = { True: 'zip-safe', -bob From robinbryce at gmail.com Thu Jul 13 21:52:03 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Thu, 13 Jul 2006 20:52:03 +0100 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy Message-ID: Hi, [using setuptools 0.6b4] I'm a setuptools user and greatly appreciative of it as well. I'd like to understand how to use it more appropriately with respect to bundling miscellaneous data files. "just put them in python packages" is really not what I want but perhaps its time to refactor my tastes and accept this is the most appropriate thing to do. I think the root of my confusion lies in how I perceive the 'pkg_' prefix of the setuptools pkg_resources api. I think of it in the general sense of the word package - data I have 'packaged' along with my library or application. Experience suggests it would be better read in the python specific sense of 'package': data that is contained in a file that is a descendant of a python package (the file must be in or under a directory that contains an __init__.py). Which of these two views most accurately reflects the usage for which pkg_resources was designed ? Is it possible to have a separate 'zip_safe' decision for data files versus python packages. Ie., a deployed egg with data files and non zip safe packages would appear in site-packages (or wherever) as both a zip archive for the zip safe data AND a directory tree containing the 'eager' resources ? As context for the rest of this plea for help: A trivial layout that bundles a .conf file:: setuptools-test lib/foopackage/__init__.py lib/foopackage/foo.py conf/foo.conf setup.py versions etc: Ubuntu(dapper), Python 2.4.3, setuptools-0.6b4. My python is installed with the prefix: /home/robin/devel/0root. It is a 'proper' install rather than virtual-python.py setup. Is there a specific reason why there isn't a find_data_files to compliment find_packages in setuptools ? eg., ``data_files=[('/', find_data_files('*.conf'))`` spells: recursively find all .conf files, starting in the directory containing my setup.py, and bundle them in my egg root. I would then expect ``unzip -l foopackage-VER-py2.4.egg`` to produce a tree like:: foopackage/*.py, *.pyc conf/foo.conf EGG-INFO/ # usual suspects Does/could setuptools to overload the distutils keyword 'data_files' and change it's meaning so that it can work with pkg_resources rather than being --prefix relative ? (package_data, while a useful 2.4 addition, is not what I want here) In foopackage/foo.py Why are all of:: pkg_resources(Requirement(__name__), '/conf/foo.conf') pkg_resources(Requirement(__name__), 'conf/foo.conf') pkg_resources(Requirement('foopackage'), '/conf/foo.conf') interpreted as relative to the foopackage directory ? And why does the resource_name '/' not refer to the top of the egg ? Irrespective of whether I specify a relative or absolute path above pkg_resources always looks under the top most package directory. Is this by design ? Non packaged data files are packaged as siblings of 'foopackage'. What is the most convenient way package and access these files such that the references work for egg installs, normal 'setup.py install' installations and for ``python setup.py develop`` pseudo installs ? Extending my example with the following changes I explore pkg_resources.resource_string and friends:: file:setup.cfg [egg_info] egg_base=./ # because I guessed (incorrectly) that this would help. file:setup.py:: from setuptools import setup, find_packages setup( name='foopackage', packages=find_packages('lib'), package_dir={'','lib'}, data_files=[('conf','conf/foo.conf')], entry_points=dict(console_scripts=[ 'fooconf = foopackage.foo:run'])) file:foopackage/foo.py:: from pkg_resources import resource_string def run(): print __name__,__file__ try: foo_config=resource_string(__name__,'/conf/foo.conf') except IOError, e: print str(e) else: print foo_config if __name__=='__main__': run() running in place:: $python setuptools-test/lib/foopackage/foo.py __main__ setuptools-test/lib/foopackage/foo.py [Errno 20] Not no such file or directory: 'setuptools-test/lib/foopackage/conf/foo.conf' This is (almost) what I'd expect, I have not run setup.py yet so setuptools/pkg_resources has no way of knowing anything about my weirdo preferences. Given that setuptools has not had a chance to see my egg_base setting, I would expect '/' to mean the directory *containing* the top most package inferable from __file__. So I would have expected the path in the error to be 'setuptools-test/lib/conf/foo.conf'. But I don't care so much about the 'pre setup.py' scenario. Make an egg:: $python setup.py bdist_egg --keep-temp copying conf/foo.conf -> build/bdist.linux-i686/egg/conf creating 'dist/foopackage-0.0.0-py2.4.egg' and adding 'build/bdist.linux-i686/egg' to it $ls build/bdist.linux-i686/egg conf EGG-INFO foopackage $unzip -l dist/foopackage-0.0.0-py2.4.egg # paraphrasing the output foopackage/ *.py *.pyc conf/foo.conf EGG-INFO/ Woot! Exactly what I had hoped for. Install the package using develop mode (note the explicit egg_base option above):: First, manually clean up site-packages just to be sure. (rm easy-install.pth; rm foopackage*) $cd setuptools-test $python setup.py develop Installing fooconf script to /home/robin/devel/0root/bin $cd .. $fooconf Traceback (most recent call last): ImportError: No module named foopackage.foo $cat $PYSITE/foopackage.egg-link /home/robin/devel/setuptools-test $cat $PYSITE/easy-install.pth import sys; sys.__plen = len(sys.path) /home/robin/devel/setuptools-test Rats. It seems like egg_base is taken both as the place to put my .egg-info directory AND as the means of deciding what should be placed on sys.path in order for my package to be importable. remembering the package_dir option I reach for the distutils docs. But a variety of package=[], package_dir=[] combinations have no effect on the easy-install.pth. Double rats. Is there a way to have easy-install.pth, in the develop case, to get entries with the form: '/path/to/source-root/my/python/packages/live/here' ^--------------------------^ this is the bit we already have | assuming egg_base=./ (which it does not by default) AND an independent way of directing pkg_resources to where my data files are rooted ?? I look at the egg install case:: delete dist & build & do python setup.py bdist_egg, delete easy-install.pth and *.link files because I've been fiddling. $easy_install dist/foopackage-0.0.0-py2.4.egg Processing foopackage-0.0.0-py2.4.egg Copying foopackage-0.0.0-py2.4.egg to /home/robin/devel/0root/lib/python2.4/site-packages Adding foopackage 0.0.0 to easy-install.pth file Installed /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Processing dependencies for foopackage==0.0.0 Erm what happened to my script ? (using setuptools 0.6b4). Quoting the easy_install docs: "Whenever you install, upgrade, or change versions of a package, EasyInstall automatically installs the scripts for the selected package version, unless you tell it not to" I fire up python:: $pwd /home/robin/devel/setuptools-test $cd .. $which python $/home/robin/devel/0root/bin/python $python >>>from foopackage.foo import run >>>run() foopackage.foo /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/foopackage/foo.pyc [Errno 2] No such file or directory: '/home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/foopackage/conf/foo.conf' Ctrl-D $ls 0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg/conf/ foo.conf (extreme gnashing of teeth, followed by a visit to the refrigerator so i can throw some eggs at the wall) I give up on egg_base, delete my setup.cfg, manually clean up my site-packages directory and delete my dist & build trees. I create a new egg, this time without the egg_base option:: $cd setuptools-test $python setup.py bdist_egg $unzip -l dist/foopackage-0.0.0-py2.4.egg # paraphrasing the output foopackage/ *.py *.pyc conf/foo.conf EGG-INFO/ Again, exactly what I want and shows that egg_base has *no* effect on the internal layout of the egg. Lets install it:: $easy_install dist/foopackage-0.0.0-py2.4.egg Processing foopackage-0.0.0-py2.4.egg creating /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Extracting foopackage-0.0.0-py2.4.egg to /home/robin/devel/0root/lib/python2.4/site-packages Adding foopackage 0.0.0 to easy-install.pth file Installing fooconf script to /home/robin/devel/0root/bin Installed /home/robin/devel/0root/lib/python2.4/site-packages/foopackage-0.0.0-py2.4.egg Processing dependencies for foopackage==0.0.0 What the frak, this time I get my script. What on earth does egg_base have to do with script generation ? I try it out:: $fooconf foopackage.foo PYSITE/foopackage-0.0.0-py2.4.egg/foopackage/foo.pyc [Errno 2] No such file or directory: 'PYSITE/foopackage-0.0.0-py2.4.egg/foopackage/conf/foo.conf' Curses. The fact that distutils does not include files specified in MANIFIEST.in for anything other than the sdist command (source distributions) is really tedious (and horribly confusing when first encountered). (minor nit) the include_package_data option is well named but poorly described: "Accept all data files and directories matched by MANIFEST.in or found in source control". This is a lie. Only those files that are desendants of a python package directory (directory that has __init__.py) are considered. "Accept all python package data files " would reduce confusion and pointless optimism. At least for those that know just enough to get themselves into trouble (like me). I very much would prefer that the machinery for including data files in a package to be orthogonal to the source building/packaging machinery. I think there is a compelling argument that says complex data should be explicitly packaged separately. Ie if foopackage had non trivial data then I, as the package author, should create and distribute foopackage.egg and foopackage.data.egg as separate things. foopackage.egg would require foopackage.data and would, as an additionally benefit, be free to use existant setuptools machinery to separate data versions from package versions. In fact, to argue completely against the thrust of this mail I'm now thinking along the lines of: - *never* package data in the same egg as the application or library - *always* create a separate foopackage-data package, even if it has no python source in it beyond setup.py and even if the data is trivial. - use the optional dependencies mechanism to pull data in as needed. Anyhow. Thats got that lot off my chest. I have no intention of giving up on setuptools, it is *far* to useful for that. I do want to hear from distutils folks that could help straighten me out :-) Cheers, Robin From jim at zope.com Thu Jul 13 21:57:45 2006 From: jim at zope.com (Jim Fulton) Date: Thu, 13 Jul 2006 15:57:45 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: On Jul 11, 2006, at 3:12 PM, Phillip J. Eby wrote: > I have just released version 0.6b4 of setuptools, the last beta > release of > setuptools 0.6. Please upgrade and test at your earliest > convenience, as I > would like to issue a release candidate version next week. With the new version, I'm still having the problem that the --no-deps option is ignored by setup.py develop. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From bob at redivi.com Thu Jul 13 22:42:56 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 13:42:56 -0700 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: References: Message-ID: On Jul 13, 2006, at 12:52 PM, Robin Bryce wrote: > [using setuptools 0.6b4] > > I'm a setuptools user and greatly appreciative of it as well. I'd like > to understand how to use it more appropriately with respect to > bundling miscellaneous data files. "just put them in python packages" > is really not what I want but perhaps its time to refactor my tastes > and accept this is the most appropriate thing to do. Refactor your tastes, it is the most appropriate thing to do. -bob From bob at redivi.com Thu Jul 13 22:52:48 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 13:52:48 -0700 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: References: Message-ID: <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> On Jul 13, 2006, at 12:52 PM, Robin Bryce wrote: > Hi, > > [using setuptools 0.6b4] > > Is it possible to have a separate 'zip_safe' decision for data files > versus python packages. Ie., a deployed egg with data files and non > zip safe packages would appear in site-packages (or wherever) as both > a zip archive for the zip safe data AND a directory tree containing > the 'eager' resources ? Make separate eggs. One for the data, one for the code. The code egg be zip_safe and the data one not. > I very much would prefer that the machinery for including data files > in a package to be orthogonal to the source building/packaging > machinery. I highly doubt that's what most people want to bother with. > I think there is a compelling argument that says complex data should > be explicitly packaged separately. Ie if foopackage had non trivial > data then I, as the package author, should create and distribute > foopackage.egg and foopackage.data.egg as separate things. > foopackage.egg would require foopackage.data and would, as an > additionally benefit, be free to use existant setuptools machinery to > separate data versions from package versions. In fact, to argue > completely against the thrust of this mail I'm now thinking along the > lines of: What compelling argument? > - *never* package data in the same egg as the application or library > - *always* create a separate foopackage-data package, even if it has > no python source in it beyond setup.py and even if the data is > trivial. > - use the optional dependencies mechanism to pull data in as needed. That's not really convenient for most people, but if that's what you want to do then go ahead. Shove the data in a foopackagedata package which goes in a separate egg and throw a trivial __init__.py in there. -bob From robinbryce at gmail.com Fri Jul 14 01:56:00 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Fri, 14 Jul 2006 00:56:00 +0100 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> References: <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> Message-ID: > Shove the data in a foopackagedata package > which goes in a separate egg and throw a trivial __init__.py in there. Yes, exactly. And I get the separate zip_safe as and added bonus. > What compelling argument? Ok, I should have said "I think there _may_ be a compelling argument. I'll have a go: In another thread, "best practices for creating eggs" Paul Moore said: >I work offline sufficiently often that not having local documentation >is frustrating. There's no standard for local docs, which is a >nuisance, and makes for an inconsistent story between different >packages, but I'd be concerned if setuptools made it more difficult to > bundle local docs. and Kevin Dangoor said: >Being able to get at a package's documentation after it's >installed would be nice. Phil said: > Note that I said above that I always put the documentation in an sdist > form; to obtain a package's source distribution, use: > > easy_install -e -b somedir arg... > A standard for how to install documentation would be great I think the motivation to always package docs in sdist and arrange for egg and sdist down loads to appear together is really a deficiency in how data,eggs & pkg_resources interact. All things considered sdist is the easiest thing for the packager: It's a few runes in MANIFEST.in followed by python setup.py sdist. But you loose the ability to reference that data in using pkg_resources. If I put my docs, data, etc in their own egg I get some of this back. Something like ``package_dir = package_dir = {'foopackage': './'}`` might be enough to fake me up a package to keep setuptools happy but if not I'll create one. In this instance I don't care about having an overly large distribution because the user has actively chosen to install the docs. But all of that is less than ideal. AFAICT, if pkg_resources supported egg root relative resource_names, I would be able to do anything I want, up to an including writing egg format extensions that do more specialist things with data packaged in *eggs*. There is a lot I would *like* to do with eggs as a distribution format, beyond packaging python source. I can't see a sane way to go about this in the absence of a consistent way to reference *all* of the data I have put in my egg. I would really like to know why pkg_resources('foopackage', '/conf/foo.conf') is not interpreted as relative to the egg root ? Robin On 13/07/06, Bob Ippolito wrote: > > On Jul 13, 2006, at 12:52 PM, Robin Bryce wrote: > > > Hi, > > > > [using setuptools 0.6b4] > > > > Is it possible to have a separate 'zip_safe' decision for data files > > versus python packages. Ie., a deployed egg with data files and non > > zip safe packages would appear in site-packages (or wherever) as both > > a zip archive for the zip safe data AND a directory tree containing > > the 'eager' resources ? > > Make separate eggs. One for the data, one for the code. The code egg > be zip_safe and the data one not. > > > I very much would prefer that the machinery for including data files > > in a package to be orthogonal to the source building/packaging > > machinery. > > I highly doubt that's what most people want to bother with. > > > I think there is a compelling argument that says complex data should > > be explicitly packaged separately. Ie if foopackage had non trivial > > data then I, as the package author, should create and distribute > > foopackage.egg and foopackage.data.egg as separate things. > > foopackage.egg would require foopackage.data and would, as an > > additionally benefit, be free to use existant setuptools machinery to > > separate data versions from package versions. In fact, to argue > > completely against the thrust of this mail I'm now thinking along the > > lines of: > > What compelling argument? > > > - *never* package data in the same egg as the application or library > > - *always* create a separate foopackage-data package, even if it has > > no python source in it beyond setup.py and even if the data is > > trivial. > > - use the optional dependencies mechanism to pull data in as needed. > > That's not really convenient for most people, but if that's what you > want to do then go ahead. Shove the data in a foopackagedata package > which goes in a separate egg and throw a trivial __init__.py in there. > > -bob > > From pje at telecommunity.com Fri Jul 14 02:21:06 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 13 Jul 2006 20:21:06 -0400 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: References: <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> Message-ID: <5.1.1.6.0.20060713201856.0202cad8@sparrow.telecommunity.com> At 12:56 AM 7/14/2006 +0100, Robin Bryce wrote: >But all of that is less than ideal. AFAICT, if pkg_resources supported >egg root relative resource_names, I would be able to do anything I >want, up to an including writing egg format extensions that do more >specialist things with data packaged in *eggs*. I don't have time right now to respond to this thread in more detail, but I just want to pop in and point out the *reason* that pkg_resources doesn't support egg root relative resource names is because it won't work right with system packaging tools like RPM, Debian, etc. You've pretty much got to use the "metadata" API if you want stuff that belongs to the project instead of to a package in the project. The metadata API has some limitations, however (i.e., no file access), but these might be able to be lifted in a future release in the 0.7 or later series. From bob at redivi.com Fri Jul 14 02:34:06 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 17:34:06 -0700 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: References: <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> Message-ID: On Jul 13, 2006, at 4:56 PM, Robin Bryce wrote: >> What compelling argument? > > Ok, I should have said "I think there _may_ be a compelling argument. > I'll have a go: > > In another thread, "best practices for creating eggs" Paul Moore said: > >> I work offline sufficiently often that not having local documentation >> is frustrating. There's no standard for local docs, which is a >> nuisance, and makes for an inconsistent story between different >> packages, but I'd be concerned if setuptools made it more >> difficult to >> bundle local docs. > > and Kevin Dangoor said: > >> Being able to get at a package's documentation after it's >> installed would be nice. > > Phil said: > >> Note that I said above that I always put the documentation in an >> sdist >> form; to obtain a package's source distribution, use: >> >> easy_install -e -b somedir arg... > >> A standard for how to install documentation would be great > > I think the motivation to always package docs in sdist and arrange for > egg and sdist down loads to appear together is really a deficiency in > how data,eggs & pkg_resources interact. All things considered sdist is > the easiest thing for the packager: It's a few runes in MANIFEST.in > followed by python setup.py sdist. But you loose the ability to > reference that data in using pkg_resources. > > If I put my docs, data, etc in their own egg I get some of this back. > Something like ``package_dir = package_dir = {'foopackage': './'}`` > might be enough to fake me up a package to keep setuptools happy but > if not I'll create one. In this instance I don't care about having an > overly large distribution because the user has actively chosen to > install the docs. You said data before, not supplementary materials like docs and examples. Data that's consumed by the code absolutely belongs in eggs and shouldn't be strewn all over the place. One of the most important features of eggs is that they consolidate stuff. Without eggs you install a package and you get files thrown all over your disk without any way to keep track of what belongs to which package. It would be nice if there were a convention for installing supplementary stuff (docs, examples, py2app/py2exe generated tools, xcode templates, etc.). One problem with this is that you generally want those things to be somewhere easily accessible by the user, and something based on site-packages isn't (or sys.prefix at all in some cases, like Mac OS X). I'm not sure how much integration with distutils this stuff really deserves because it's not required to make anything work. For now, the author could just provide a zip of the supplementary materials for the users to download if they want to look at them offline. Another thing that setuptools is currently missing is support for packaging up dynamic libraries (e.g. pygame's SDL dlls) and/or headers (e.g. Numeric's API), but that's a bigger distutils problem. > But all of that is less than ideal. AFAICT, if pkg_resources supported > egg root relative resource_names, I would be able to do anything I > want, up to an including writing egg format extensions that do more > specialist things with data packaged in *eggs*. Egg format extensions are already provided for by entry points and the EGG-INFO metadata dir. Eggs were designed to facilitate plug-ins, after all. -bob From robinbryce at gmail.com Fri Jul 14 04:19:25 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Fri, 14 Jul 2006 03:19:25 +0100 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: References: <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> Message-ID: > Data that's consumed by the code absolutely belongs in eggs > and shouldn't be strewn all over the place Totally agree. even if this means I've contradicted my self some where along the line. > the *reason* that pkg_resources doesn't > support egg root relative resource names is because it won't work right > with system packaging tools like RPM, Debian, etc. ah. food for thought. thanks. > is currently missing is support for packaging up dynamic libraries Oh, I've been to timid to even consider trying that. But mixed language development needs is something I very nearly mentioned in last post. I had games development in mind[1]. I just can't see how far it is reasonable to expect 'stock' egg/setuptools to stretch. I mean where would it all end: pydselect, pje nervous break down ? > Egg format extensions are already provided for by entry points and > the EGG-INFO metadata dir So if I decide I really care about referencing data in this way: investigating adding setup keywords, files to EGG-INFO and inventing my own rules is the way to go ? Hrm, time to pull down the the setuptools trunk I think ;-) Thanks, Robin [1] The small number of uk games developers I've worked for use lua and C++. And wouldn't dream of using a packaging system they did not write them selves. From ronaldoussoren at mac.com Fri Jul 14 07:33:55 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 14 Jul 2006 07:33:55 +0200 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: References: <14677095-FEA2-48F2-9B90-7CC03FC45898@redivi.com> Message-ID: On Jul 14, 2006, at 1:56 AM, Robin Bryce wrote: >> Shove the data in a foopackagedata package >> which goes in a separate egg and throw a trivial __init__.py in >> there. > > Yes, exactly. And I get the separate zip_safe as and added bonus. > >> What compelling argument? > > Ok, I should have said "I think there _may_ be a compelling argument. > I'll have a go: > > In another thread, "best practices for creating eggs" Paul Moore said: > >> I work offline sufficiently often that not having local documentation >> is frustrating. There's no standard for local docs, which is a >> nuisance, and makes for an inconsistent story between different >> packages, but I'd be concerned if setuptools made it more >> difficult to >> bundle local docs. Documentation can be tweaked into the current setup, with some minor changes to IDEs. Someone just has to come up with a cannonical way to do documentation eggs, then IDEs can be changed to be use this. BTW. I'm using IDE very loosely here, there could just as easy be a script that updates a index.html that people have listed in their browser bookmarks. Stuff for tools outside of the python world would be harder, PyObjC includes some templates for Xcode (Apple's IDE) and those must be installed in a specific location. Ronald From robinbryce at gmail.com Fri Jul 14 17:13:44 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Fri, 14 Jul 2006 16:13:44 +0100 Subject: [Distutils] eggs and PYTHONPATH (cont'd from peak list...) In-Reply-To: References: Message-ID: > It seems that setuptools redefines the intention of PYTHONPATH from > being a "runtime path determination" variable to an "installation > path determination" variable; why repurpose PYTHONPATH in this way, I've always seen it like this: * PYTHONPATH is determined by the shell environment * .pth files are determined at install time and or (re)configuration time. * .pth files are merged into PYTHONPATH & sys.path at interpreter startup * sys.path is runtime determined And assumed: the only programs that 'casually' do os.getenv('PYTHONPATH') are shell scripts authored in python rather than bash. And as such are on PATH. I missed the start of this thread (on other list) but it looks like setuptools *has* to get involved in the last three points. It can not know the results of non egg package startup/runtime sys.path configuration. It is adding .pth files and manipulating sys.path in order to do its version dance. setuptools has to pick a precedence for everything, versioned eggs, versioned setuptools distributions, plain old python packages found on sys.path. It looks to me more like setuptools forces version -1.0.0 on everything that is not an egg, including things identified by a customised PYTHONPATH. Phillip said: > > Eggs always override non-eggs, because eggs can be controlled in > > ways that non-egg paths can't. This sounds like a choice not a reason. Isn't it just as reasonable to say PYTHONPATH can be controlled in ways egg paths can't. I guess I'm lining up with mike here. On the face of it, it does not seem reasonable for setuptools to 'guess' the version of a plain python package on sys.path or otherwise subvert the traditional "use this package damn it" behaviour of PYTHONPATH. But I don't understand this stuff enough to state it more strongly than that. Mike said: > I would propose the order be this: > > eggs on PYTHONPATH > non-eggs on PYTHONPATH > eggs in site-packages > non-eggs in site-packages > However, I cant think of a possible scenario where a path would explicitly > exist in PYTHONPATH, non-egg or egg, where the user would still like the > system-wide installation to take precedence I can't either. mike said: > For example, if a particular package on a multi-user shell account is > upgraded systemwide from its previously non-egg version to a > setuptools/egg version, then any users who have applications running > on a custom PYTHONPATH setup with dependencies on their locally- > installed, non-egg versions will now have a broken install; theyd > have to log in, read all the setuptools docs to figure out whats > going on, and do the "virtual python" setup in order to fix things Is this all from the perspective of resolving versions using setuptools ? If so I think: - It makes no sense for the system vendor to start using eggs and *not* also bundle setuptools in this format. - all non setuptools python packages are effectively *not versioned* on this system - The system vendor is making a pretty fundamental change to there distribution versioning / packaging mechanics in doing this. They should not be doing this as an interim update to an existing release. - I always expect my scripts custom config to be broken by distro upgrade. Are there any python packages widely used by non technical users that direct their users to perform these sorts of customisations to PYTHONPATH ? (I'm assuming egg.link aka develop is an 'egg' package wrt this discussion and so just works tm) But I think this issue is distinct from how non setuptools / non version packages interact with the setuptools world. Things will look very different when: setuptools becomes part of the python standard distribution AND system vendors catch up with that release of python. Cheers, Robin From pje at telecommunity.com Sat Jul 15 01:17:30 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 14 Jul 2006 19:17:30 -0400 Subject: [Distutils] Mystery solved In-Reply-To: References: <5.1.1.6.0.20060711133050.01ef0870@sparrow.telecommunity.com> <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710174120.03df1400@sparrow.telecommunity.com> <5.1.1.6.0.20060710180631.01ee6a78@sparrow.telecommunity.com> <5.1.1.6.0.20060711110603.03d00468@sparrow.telecommunity.com> <5.1.1.6.0.20060711133050.01ef0870@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060714191209.026699e0@sparrow.telecommunity.com> At 03:56 PM 7/11/2006 -0400, Jim Fulton wrote: >On Jul 11, 2006, at 2:07 PM, Phillip J. Eby wrote: >>At 11:50 AM 7/11/2006 -0400, Jim Fulton wrote: >>>I would stop when a result is found. >> >>Even so, this means O(N x M) web hits, where N is the number of >>packages and M is the number of --find-links (including dependency >>links supplied by eggs installed so far). I don't think it's >>reasonable to hit so many non-existent URLs on non-index servers, >>and is impolite to the servers' operators. (For example, if they >>receive a daily report of all 404 errors from their web servers, as >>I do. This is pretty common on Red Hat boxes using logwatch, for >>example.) >> >>It's particularly unfair since using e.g. http:// >>peak.telecommunity.com/snapshots/ as a --find-links while >>installing, say TurboGears, would cause a whole host of "index" >>hits to subdirectories of that URL, even though none of them can or >>will be found. >> >>The fallout from this approach is far worse than any "screen >>scraping" issues we've had. > >Isn't this the approach that's followed now? No; only the --find-links pages themselves are read, and one assumes that they actually exist. :) > Aren't all of the find- links searched as well as the index? I suppose > you're referring to >the search for /projectname, which potentially doubles the number of >requests. Doubling is only the beginning. If there are 5 dependencies, or 5 requirements on the command line, then it quintuples the number of requests, and they're all going to be retrieving non-existent URLs, except for whichever link was actually the package index. Of course, this is also ignoring the UI reason why the index URL and find-links URLs are specified separately, and that is that the common case is to use PyPI and maybe also a find-link or two. If they were specified by the same option, then any use of find-links would require you to retype the index URL. So, it's not a very convenient UI to merge the concepts, as well as being neither efficient for retrieval speed nor polite to site operators. From pje at telecommunity.com Sat Jul 15 01:24:45 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 14 Jul 2006 19:24:45 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <1C91F591-0072-4E7C-9E5B-F1DC0F79FA3D@redivi.com> References: <8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com> Message-ID: <5.1.1.6.0.20060714192320.03900288@sparrow.telecommunity.com> At 09:09 PM 7/12/2006 -0700, Bob Ippolito wrote: >On Jul 12, 2006, at 9:00 PM, Bob Ippolito wrote: >>On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote: >>>I have just released version 0.6b4 of setuptools, the last beta >>>release of >>>setuptools 0.6. Please upgrade and test at your earliest >>>convenience, as I >>>would like to issue a release candidate version next week. >> >>It seems there's another bug in 0.6b4. I don't know what conditions >>cause it, however: >> >> File "/Volumes/Data/developer-external/py2app/setuptools-0.6b4- >>py2.4.egg/setuptools/dist.py", line 274, in fetch_build_egg >>AttributeError: Distribution instance has no attribute >>'dependency_links' >> >>It looks like there's no default for dependency_links in some cases.. >>so that needs to change to a getattr, or it needs to set some default. > >Ok, it seems this happens whenever you do develop, install, etc. when >all dependencies aren't available. easy_install seems to work though. As it turns out, it's due to the hairy problem of bootstrapping arguments that depend on entry points, before the requirements that initialize the entry points are initialized. I'm fixing it at the point of initialization instead of downstream, since then any new code that needs to access self.dependency_links won't end up with the same bug. From pje at telecommunity.com Sat Jul 15 01:47:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 14 Jul 2006 19:47:38 -0400 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: Message-ID: <5.1.1.6.0.20060714193925.038969b8@sparrow.telecommunity.com> At 08:52 PM 7/13/2006 +0100, Robin Bryce wrote: >file:setup.cfg >[egg_info] >egg_base=./ # because I guessed (incorrectly) that this would help. This option controls where the .egg-info *build* directory will be, and *nothing* else. But I think you figured that out. :) Meanwhile, I think the solution to your overall problem will require either expanding the metadata API to allow file access, or else adding some kind of path facility to the resource API (e.g. '/'-prefixed paths as you suggested) to access a 'data/' subdirectory of the egg-info directory. I'll look into this option in setuptools 0.7, although I think it will also require some work on the install_data command. There's also a potentially significant limitation in that you won't be able to look up non-package resources using package names. That is, you're not going to be able to do resource_filename('foo.bar', '/baz'). Instead, you would have to do resource_filename(Requirement.parse('MyProject'), '/baz'). Again, this is an unfortunate quirk caused by the need to support legacy package layouts for system packaging tools such as RPM, Debian, etc. From pje at telecommunity.com Sat Jul 15 01:49:52 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 14 Jul 2006 19:49:52 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060714193206.02c75f80@sparrow.telecommunity.com> At 03:57 PM 7/13/2006 -0400, Jim Fulton wrote: >On Jul 11, 2006, at 3:12 PM, Phillip J. Eby wrote: > >>I have just released version 0.6b4 of setuptools, the last beta >>release of >>setuptools 0.6. Please upgrade and test at your earliest >>convenience, as I >>would like to issue a release candidate version next week. > >With the new version, I'm still having the problem that the --no-deps >option is ignored by setup.py develop. That's because I never fixed it. :) Apparently I didn't get it on my list. I've checked in a fix now, along with fixes for the problems Bob found. From robinbryce at gmail.com Sat Jul 15 15:00:58 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Sat, 15 Jul 2006 14:00:58 +0100 Subject: [Distutils] distutils data_files and setuptools.pkg_resources are driving me crazy In-Reply-To: <5.1.1.6.0.20060714193925.038969b8@sparrow.telecommunity.com> References: <5.1.1.6.0.20060714193925.038969b8@sparrow.telecommunity.com> Message-ID: > This option controls where the .egg-info *build* directory will be and > *nothing* else Yes. :-) Did you notice, in my unfortunately long initial post, the issue I had with script generation when the build directory was in a non default location ? egg_base=./ seemed to prevent the installation of console_scripts. I'm planing on spending today attempting a test case patch to show this. You'd save me some time if you could say "this is not a bug" :-) [apologies, quoting you in reverse order here] > I'll look into this option in setuptools 0.7, although I think it will also > require some work on the install_data command. There's also a potentially > significant limitation in that you won't be able to look up non-package > resources using package names. That is, you're not going to be able to do > resource_filename('foo.bar', '/baz'). Instead, you would have to do > resource_filename(Requirement.parse('MyProject'), '/baz'). I don't think this is a limitation. As the package author I have gone to some trouble to say "this data is not package data". I would not then expect (or want) to be able to reference that data using the ``package`` bit of ``resouce_string(package_or_requirement, resource_name)``. If resource_name referes to non package data then I *expect* Requirement.parse('MyProject') to be its name space when referenced by my program. > Meanwhile, I think the solution to your overall problem will require either > expanding the metadata API to allow file access, or else adding some kind > of path facility to the resource API (e.g. '/'-prefixed paths as you > suggested) to access a 'data/' subdirectory of the egg-info directory. Yes, I agree. I'll have a stab at thinking this through: (Aside: I think it's a *good* thing that eggs don't attempt to be a 'full blown' virtualised file system - that would involve too much magic for my tastes, refactored or not ;-). There are at least two 'file' systems in play here: the package authors development source tree and the eggs collection of resource_names. All information to produce the latter is available, or could be available, to setuptools once ``python setup.py egg_info`` has run. A key advantage (for me at least) of ``python setup.py develop`` is that it produces the .egg-info directory but does *not* trigger a full build of my sources. Specifically, it is most advantageous where my source tree has a lot of data that would otherwise be copied into the build tree. Being able to avoid this copy and still run something that is pretty close to what my end users will run is *very* cool. It would be a shame to loose this for non packaged data files. Which suggests that .egg-info would need to remember enough information about the source tree in order to allow pkg_resources to perform a reverse mapping for resource_filename(Requirement.parse('MyProject'), '/baz') I can not see a reasonable way to support: python lib/MyProject/entrypoint.py --list-non-packaged-data. Ie, expecting pkg_resources to perform this reverse mapping *before* .egg-info is built. If python setup.py develop does *not* copy data_files into place, I don't think it's unreasonable for pkg_resources to assume EGG-INFO is always available. For all 'installed' cases I think setuptools can get at EGG-INFO without guess work. For python setup.py develop, I think heuristics in pkg_resources, that look for .egg-info relative to MyPackage are acceptable. boost bjam does something allong these lines when it looks for boost-build.jam. The confusion I had initially with egg_base was that I guessed (incorrectly) that it could be used as a way to preserve the 'prefix' information and make it available in the resultant .egg-info. I find my self *really* wanting to help get this into 0.7. I have both the time and the need to work on this. I'm happy to have a stab blind and contribute the results. Alternately, if you spec it, I'd have no trouble letting you direct my typing fingers :-) Cheers, Robin From robinbryce at gmail.com Sat Jul 15 15:51:45 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Sat, 15 Jul 2006 14:51:45 +0100 Subject: [Distutils] Keep downloaded files in case of an error In-Reply-To: <44B38A61.5050008@hlabs.spb.ru> References: <44B38A61.5050008@hlabs.spb.ru> Message-ID: python setup.py develop --build-directory=build worked for me:: robin at rsb2-ubuntu:~/devel/setuptools-test$ python setup.py develop -b build running develop running egg_info writing requirements to lib/foopackage.egg-info/requires.txt writing lib/foopackage.egg-info/PKG-INFO writing top-level names to lib/foopackage.egg-info/top_level.txt writing entry points to lib/foopackage.egg-info/entry_points.txt reading manifest file 'lib/foopackage.egg-info/SOURCES.txt' writing manifest file 'lib/foopackage.egg-info/SOURCES.txt' running build_ext Creating /home/robin/devel/0root/lib/python2.4/site-packages/foopackage.egg-link (link to lib) foopackage 0.0.0 is already the active version in easy-install.pth Installing fooconf script to /home/robin/devel/0root/bin Installed /home/robin/devel/setuptools-test/lib Processing dependencies for foopackage==0.0.0 Searching for Twisted>=2.4.0 Reading http://www.python.org/pypi/Twisted/ Reading http://twistedmatrix.com/ Reading http://www.python.org/pypi/Twisted/2.4.0 Best match: Twisted 2.4.0 Downloading http://tmrc.mit.edu/mirror/twisted/Twisted/2.4/Twisted-2.4.0.tar.bz2Processing Twisted-2.4.0.tar.bz2 Running setup.py -q bdist_egg --dist-dir build/twisted/egg-dist-tmp-n9yA5h usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: invalid command 'bdist_egg' Error: Subprocess exited with result 1 for project core error: Setup script exited with 1 robin at rsb2-ubuntu:~/devel/setuptools-test$ ls build/twisted LICENSE TwistedConch-0.7.0 TwistedNames-0.3.0 TwistedWords-0.4.0 README TwistedCore-2.4.0 TwistedNews-0.2.0 ZopeInterface-3.1.0c1 setup.py TwistedLore-0.2.0 TwistedRunner-0.2.0 temp TwistedMail-0.3.0 TwistedWeb-0.6.0 and:: robin at rsb2-ubuntu:~/devel/setuptools-test$ python setup.py develop --help --build-directory (-b) download/extract/build in DIR; keep the results From robinbryce at gmail.com Sat Jul 15 16:22:50 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Sat, 15 Jul 2006 15:22:50 +0100 Subject: [Distutils] md5 checksums through PyPI XML-RPC interface In-Reply-To: <1F6714AF-55F7-4D78-AA54-4685A96B8DCA@mac.com> References: <1F6714AF-55F7-4D78-AA54-4685A96B8DCA@mac.com> Message-ID: After quick look I couln't find more than: http://mail.python.org/pipermail/catalog-sig/2005-June/000656.html on PyPI XML-RPC support. That mail does not mention md5. From: '[Distutils] "Package Index API" draft' http://mail.python.org/pipermail/distutils-sig/2006-July/006496.html > In addition, for compatibility with PyPI versions that do not use ``#md5=`` > fragment IDs, EasyInstall uses the following regular expression to match > PyPI's displayed MD5 info (broken onto two lines for readability):: > > ([^<]+)\n\s+\( &digest=([0-9a-f]{32})">md5\) I *think* this suggest that for a full implementation you have no choice other than rummaging around in the html. Robin On 06/07/06, Ronald Oussoren wrote: > Hi, > > I'm building a tool that interfaces with PyPI and noticed that while > it is possible to fetch a lot of information through XML-RPC, the MD5 > checksum of files isn't amongst that. > > Ronald > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > > > From robinbryce at gmail.com Sat Jul 15 23:06:09 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Sat, 15 Jul 2006 22:06:09 +0100 Subject: [Distutils] [PATCH] catch egg_base=./ foolishness ? was (distutils data_files and setuptools.pkg_resources are driving me crazy) Message-ID: On 15/07/06, Robin Bryce wrote: > egg_base=./ seemed to prevent the installation of console_scripts. Oh fiddle sticks. this is because './' is not equal to os.curdir on *any* system. line 103: egg_info.py does the right thing: if self.egg_base != os.curdir. This mirrors exactly how distutils.filelist.findall does the same gaurd on files in the current directory. Observing that: os.path.join('.' + os.sep, '') == os.path.join(os.curdir, '') and os.path.join('.', '') == os.path.join(os.curdir, '') how about this patch:: Index: setuptools/command/egg_info.py =================================================================== --- setuptools/command/egg_info.py (revision 85) +++ setuptools/command/egg_info.py (working copy) @@ -100,7 +100,7 @@ self.ensure_dirname('egg_base') self.egg_info = to_filename(self.egg_name)+'.egg-info' - if self.egg_base != os.curdir: + if os.path.join(self.egg_base, '') != os.path.join(os.curdir,''): self.egg_info = os.path.join(self.egg_base, self.egg_info) if '-' in self.egg_name: self.check_broken_egg_info() From bob at redivi.com Mon Jul 17 09:26:28 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 00:26:28 -0700 Subject: [Distutils] setuptools and extra_path Message-ID: <0ECDEA76-36A4-411D-9279-EE1B93EF5042@redivi.com> Is there any reason why setuptools totally ignores extra_path when using compatibility mode (e.g. calling .run() manually or --single- version-externally-managed)? It makes uninstallation much more painful than it would've been otherwise. -bob From lists at hlabs.spb.ru Mon Jul 17 14:52:18 2006 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Mon, 17 Jul 2006 16:52:18 +0400 Subject: [Distutils] Keep downloaded files in case of an error In-Reply-To: References: <44B38A61.5050008@hlabs.spb.ru> Message-ID: <44BB8802.2050704@hlabs.spb.ru> Robin Bryce wrote: > python setup.py develop --build-directory=build worked for me:: I know about "--build-directory" option but it doesn't work in case of "unknown archive type" error for example: $ python setup.py develop -b build ... Reading http://www.python.org/pypi/Twisted/ Reading http://twistedmatrix.com/ Best match: Twisted 2.4.0 Downloading http://tmrc.mit.edu/mirror/twisted/Twisted/2.4/Twisted-2.4.0.tar.bz2 Processing Twisted-2.4.0.tar.bz2 error: Not a recognized archive type: /tmp/easy_install-poUijE/Twisted-2.4.0.tar.bz2 $ ls -l build ls: build: No such file or directory I guess downloaded file is created as temporary file so in case of a such error it's just discarded. -- Dmitry Vasiliev (dima at hlabs.spb.ru) http://hlabs.spb.ru From bob at redivi.com Mon Jul 17 16:17:18 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 07:17:18 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: <32A53630-2832-4FD0-8752-5E15B4F4A4FE@redivi.com> On Jul 11, 2006, at 12:33 PM, Bob Ippolito wrote: > > On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote: > >> Last, but not least, the ability was added to turn off SVN revision >> numbers >> or dates from the command line, so that you don't have to edit >> setup.cfg in >> order to issue a release. > > Would've been convenient if you said what the option was... took me a > few minutes to figure out where to look. Presumably you're referring > to the --no-svn-revision option? It seems that --no-svn-revision is only part of the solution to making releases less of a hassle. The problem is that setup.cfg is still included in sdist, so any user that builds your egg from source is going to have it tagged with ".dev_r0". This is especially problematic for dependencies because it may download the correct version of source, but ".dev_r0" ranks lower so it will say the dependency is not satisfied. What's the best solution to this, barring deleting setup.cfg from the release branch? -bob From pje at telecommunity.com Mon Jul 17 23:30:31 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Jul 2006 17:30:31 -0400 Subject: [Distutils] [PATCH] catch egg_base=./ foolishness ? was (distutils data_files and setuptools.pkg_resources are driving me crazy) In-Reply-To: Message-ID: <5.1.1.6.0.20060717172709.0205b0e8@sparrow.telecommunity.com> At 10:06 PM 7/15/2006 +0100, Robin Bryce wrote: >On 15/07/06, Robin Bryce wrote: >>egg_base=./ seemed to prevent the installation of console_scripts. > >Oh fiddle sticks. this is because './' is not equal to os.curdir on >*any* system. > >line 103: egg_info.py does the right thing: if self.egg_base != >os.curdir. This mirrors exactly how distutils.filelist.findall does >the same gaurd on files in the current directory. > >Observing that: >os.path.join('.' + os.sep, '') == os.path.join(os.curdir, '') and >os.path.join('.', '') == os.path.join(os.curdir, '') > >how about this patch:: I don't see the point of the patch. egg_base is supposed to be a directory name. './' is not a directory name, and I don't see a use case for setting egg_base to that. In fact, 99% or more of the time, there's no use case for setting --egg-base in the first place. The option was created mainly to support certain kinds of workaround scenarios in the early days of setuptools development. I would deprecate the option, except that it doesn't really do any harm, unless somebody messes around with it on the theory that it does more than what it actually does. :) From pje at telecommunity.com Mon Jul 17 23:33:02 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Jul 2006 17:33:02 -0400 Subject: [Distutils] setuptools and extra_path In-Reply-To: <0ECDEA76-36A4-411D-9279-EE1B93EF5042@redivi.com> Message-ID: <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> At 12:26 AM 7/17/2006 -0700, Bob Ippolito wrote: >Is there any reason why setuptools totally ignores extra_path when >using compatibility mode (e.g. calling .run() manually or --single- >version-externally-managed)? It's probably an unintended side effect of the fact that setuptools doesn't support extra_path. > It makes uninstallation much more painful than it would've been otherwise. Weren't you using --root or --record? You pretty much have to do one or the other for backward compatibility to be in effect, I believe. From pje at telecommunity.com Mon Jul 17 23:35:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Jul 2006 17:35:32 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <32A53630-2832-4FD0-8752-5E15B4F4A4FE@redivi.com> References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> At 07:17 AM 7/17/2006 -0700, Bob Ippolito wrote: >It seems that --no-svn-revision is only part of the solution to >making releases less of a hassle. The problem is that setup.cfg is >still included in sdist, so any user that builds your egg from source >is going to have it tagged with ".dev_r0". This is especially >problematic for dependencies because it may download the correct >version of source, but ".dev_r0" ranks lower so it will say the >dependency is not satisfied. > >What's the best solution to this, barring deleting setup.cfg from the >release branch? Argh. Well, you could put an exclude in MANIFEST.in to drop the setup.cfg, but that sucks. Maybe the right thing to do is to include the setup.cfg, but to update it with the tag and version settings that were in effect when the sdist was built. From bob at redivi.com Mon Jul 17 23:42:54 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 14:42:54 -0700 Subject: [Distutils] [PATCH] catch egg_base=./ foolishness ? was (distutils data_files and setuptools.pkg_resources are driving me crazy) In-Reply-To: <5.1.1.6.0.20060717172709.0205b0e8@sparrow.telecommunity.com> References: <5.1.1.6.0.20060717172709.0205b0e8@sparrow.telecommunity.com> Message-ID: On Jul 17, 2006, at 2:30 PM, Phillip J. Eby wrote: > At 10:06 PM 7/15/2006 +0100, Robin Bryce wrote: >> On 15/07/06, Robin Bryce wrote: >>> egg_base=./ seemed to prevent the installation of console_scripts. >> >> Oh fiddle sticks. this is because './' is not equal to os.curdir on >> *any* system. >> >> line 103: egg_info.py does the right thing: if self.egg_base != >> os.curdir. This mirrors exactly how distutils.filelist.findall does >> the same gaurd on files in the current directory. >> >> Observing that: >> os.path.join('.' + os.sep, '') == os.path.join(os.curdir, '') and >> os.path.join('.', '') == os.path.join(os.curdir, '') >> >> how about this patch:: > > I don't see the point of the patch. egg_base is supposed to be a > directory > name. './' is not a directory name, and I don't see a use case for > setting > egg_base to that. In fact, 99% or more of the time, there's no use > case > for setting --egg-base in the first place. The option was created > mainly > to support certain kinds of workaround scenarios in the early days of > setuptools development. I would deprecate the option, except that it > doesn't really do any harm, unless somebody messes around with it > on the > theory that it does more than what it actually does. :) FYI egg_base what we're using on the PyObjC trunk [1] in order to facilitate "python setup.py develop" support without re-organizing everything. All of the top-level Python packages are in a Lib directory, so we set egg_base="Lib". [1] http://svn.red-bean.com/pyobjc/trunk/pyobjc/ -bob From bob at redivi.com Mon Jul 17 23:47:30 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 14:47:30 -0700 Subject: [Distutils] setuptools and extra_path In-Reply-To: <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> References: <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> Message-ID: <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> On Jul 17, 2006, at 2:33 PM, Phillip J. Eby wrote: > At 12:26 AM 7/17/2006 -0700, Bob Ippolito wrote: >> Is there any reason why setuptools totally ignores extra_path when >> using compatibility mode (e.g. calling .run() manually or --single- >> version-externally-managed)? > > It's probably an unintended side effect of the fact that setuptools > doesn't support extra_path. Why not support it with --single-version-externally-managed? >> It makes uninstallation much more painful than it would've been >> otherwise. > > Weren't you using --root or --record? You pretty much have to do > one or the other for backward compatibility to be in effect, I > believe. No, but it's a subclass of the install command. It's caught by this in setuptools.command.install.install.run: caller = sys._getframe(2) caller_module = caller.f_globals.get('__name__','') caller_name = caller.f_code.co_name if caller_module != 'distutils.dist' or caller_name! ='run_commands': # We weren't called from the command line or setup(), so we # should run in backward-compatibility mode to support bdist_* # commands. _install.run(self) -bob From bob at redivi.com Mon Jul 17 23:49:31 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 14:49:31 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> Message-ID: <05D15BCF-8880-4DA1-99E0-52FE69EA1DB2@redivi.com> On Jul 17, 2006, at 2:35 PM, Phillip J. Eby wrote: > At 07:17 AM 7/17/2006 -0700, Bob Ippolito wrote: >> It seems that --no-svn-revision is only part of the solution to >> making releases less of a hassle. The problem is that setup.cfg is >> still included in sdist, so any user that builds your egg from source >> is going to have it tagged with ".dev_r0". This is especially >> problematic for dependencies because it may download the correct >> version of source, but ".dev_r0" ranks lower so it will say the >> dependency is not satisfied. >> >> What's the best solution to this, barring deleting setup.cfg from the >> release branch? > > Argh. Well, you could put an exclude in MANIFEST.in to drop the > setup.cfg, but that sucks. Maybe the right thing to do is to > include the setup.cfg, but to update it with the tag and version > settings that were in effect when the sdist was built. That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision). Any chance of getting this in setuptools 0.6 or should I start adding MANIFEST.in files to the relevant projects? -bob From bob at redivi.com Tue Jul 18 00:07:48 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 15:07:48 -0700 Subject: [Distutils] setuptools and extra_path In-Reply-To: <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> References: <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> Message-ID: <8C0D93E9-2FB3-4932-88E4-DD4A927FFA46@redivi.com> On Jul 17, 2006, at 2:47 PM, Bob Ippolito wrote: > > On Jul 17, 2006, at 2:33 PM, Phillip J. Eby wrote: > >> At 12:26 AM 7/17/2006 -0700, Bob Ippolito wrote: >>> Is there any reason why setuptools totally ignores extra_path when >>> using compatibility mode (e.g. calling .run() manually or --single- >>> version-externally-managed)? >> >> It's probably an unintended side effect of the fact that setuptools >> doesn't support extra_path. > > Why not support it with --single-version-externally-managed? And further, it should probably issue some kind of warning to let people know that setuptools is ignoring this option in any scenario when it is ignored... I didn't even notice until I started looking in my site-packages folder to uninstall stuff. In this particular case I don't need --root or --record because I already am tracking all of that with bdist_mpkg in a different way. The reason I want single-version-externally-managed behavior in the first place is because I don't want to have to try and deal with having the packages try and edit easy_install.pth. I suppose I could have it install in an egg-dir and drop in a separate pth, but ideally I wouldn't have to do anything. All I want is the old behavior plus the egg_info metadata. There are three features bdist_mpkg has that make it currently useful despite easy_install: [1] No bootstrap issue, Mac OS X ships with a GUI installer that understands .mpkg/.pkg bundles. [2] No permission issue. Mac OS X will prompt for authorization (like "sudo") when and only when it is needed [3] Installing Other Stuff. bdist_mpkg is used to install examples, documentation, headers, dependent libraries, etc. The third feature is important in a few cases, because packages like pygame have dependent libraries (the SDL frameworks), PyObjC has Xcode templates, etc. Most packages are perfectly well suited to eggs as currently specified, though. I'm not too concerned with the other two features, because those are rarely an issue and can be easily resolved with a GUI front-end to easy_install that combines ez_setup- like functionality for bootstrapping. -bob From pje at telecommunity.com Tue Jul 18 01:08:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Jul 2006 19:08:38 -0400 Subject: [Distutils] [PATCH] catch egg_base=./ foolishness ? was (distutils data_files and setuptools.pkg_resources are driving me crazy) In-Reply-To: References: <5.1.1.6.0.20060717172709.0205b0e8@sparrow.telecommunity.com> <5.1.1.6.0.20060717172709.0205b0e8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060717190640.0205cbc8@sparrow.telecommunity.com> At 02:42 PM 7/17/2006 -0700, Bob Ippolito wrote: >On Jul 17, 2006, at 2:30 PM, Phillip J. Eby wrote: > >>At 10:06 PM 7/15/2006 +0100, Robin Bryce wrote: >>>On 15/07/06, Robin Bryce wrote: >>>>egg_base=./ seemed to prevent the installation of console_scripts. >>> >>>Oh fiddle sticks. this is because './' is not equal to os.curdir on >>>*any* system. >>> >>>line 103: egg_info.py does the right thing: if self.egg_base != >>>os.curdir. This mirrors exactly how distutils.filelist.findall does >>>the same gaurd on files in the current directory. >>> >>>Observing that: >>>os.path.join('.' + os.sep, '') == os.path.join(os.curdir, '') and >>>os.path.join('.', '') == os.path.join(os.curdir, '') >>> >>>how about this patch:: >> >>I don't see the point of the patch. egg_base is supposed to be a >>directory >>name. './' is not a directory name, and I don't see a use case for >>setting >>egg_base to that. In fact, 99% or more of the time, there's no use >>case >>for setting --egg-base in the first place. The option was created >>mainly >>to support certain kinds of workaround scenarios in the early days of >>setuptools development. I would deprecate the option, except that it >>doesn't really do any harm, unless somebody messes around with it >>on the >>theory that it does more than what it actually does. :) > >FYI egg_base what we're using on the PyObjC trunk [1] in order to >facilitate "python setup.py develop" support without re-organizing >everything. All of the top-level Python packages are in a Lib >directory, so we set egg_base="Lib". If your package_dir={'':'Lib'}, the egg_base should be set automatically. This should also eliminate the need for the more-complex package_dir you're currently using. From pje at telecommunity.com Tue Jul 18 01:10:09 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Jul 2006 19:10:09 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <05D15BCF-8880-4DA1-99E0-52FE69EA1DB2@redivi.com> References: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060717190924.02447288@sparrow.telecommunity.com> At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: >On Jul 17, 2006, at 2:35 PM, Phillip J. Eby wrote: >>Argh. Well, you could put an exclude in MANIFEST.in to drop the >>setup.cfg, but that sucks. Maybe the right thing to do is to >>include the setup.cfg, but to update it with the tag and version >>settings that were in effect when the sdist was built. > >That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision). >Any chance of getting this in setuptools 0.6 or should I >start adding MANIFEST.in files to the relevant projects? That depends on how big the change ends up being. If it looks simple and unlikely to fail, I could live with adding it to 0.6. From pje at telecommunity.com Tue Jul 18 01:17:14 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Jul 2006 19:17:14 -0400 Subject: [Distutils] setuptools and extra_path In-Reply-To: <8C0D93E9-2FB3-4932-88E4-DD4A927FFA46@redivi.com> References: <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> Message-ID: <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> At 03:07 PM 7/17/2006 -0700, Bob Ippolito wrote: >On Jul 17, 2006, at 2:47 PM, Bob Ippolito wrote: > >> >>On Jul 17, 2006, at 2:33 PM, Phillip J. Eby wrote: >> >>>At 12:26 AM 7/17/2006 -0700, Bob Ippolito wrote: >>>>Is there any reason why setuptools totally ignores extra_path when >>>>using compatibility mode (e.g. calling .run() manually or --single- >>>>version-externally-managed)? >>> >>>It's probably an unintended side effect of the fact that setuptools >>>doesn't support extra_path. >> >>Why not support it with --single-version-externally-managed? > >And further, it should probably issue some kind of warning to let >people know that setuptools is ignoring this option in any scenario >when it is ignored... I didn't even notice until I started looking in >my site-packages folder to uninstall stuff. > >In this particular case I don't need --root or --record because I >already am tracking all of that with bdist_mpkg in a different way. > >The reason I want single-version-externally-managed behavior in the >first place is because I don't want to have to try and deal with >having the packages try and edit easy_install.pth. I suppose I could >have it install in an egg-dir and drop in a separate pth, but ideally >I wouldn't have to do anything. All I want is the old behavior plus >the egg_info metadata. Erm, ok, but the old behavior creates a .pth file, so you're kind of confusing me here. The other weird bit is that install_egg_info would need to install to the extra_path directory. Oy. And that's not the worst of it. "install" computes its target paths during finalize_options(), so there's no way to know at that point whether run() is going to be called in backward-compatibility mode. So, there doesn't seem to be a way to implement extra_path for backward-compatibility mode that doesn't set the --root or --single-version-externally-managed options. If you can live with that limitation (explicitly setting one or both of those options when making a super call), I can have it respect extra_path. But it's a no-go in any other circumstance, I'm afraid. From bob at redivi.com Tue Jul 18 01:29:28 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 16:29:28 -0700 Subject: [Distutils] setuptools and extra_path In-Reply-To: <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> References: <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> Message-ID: <75E05796-F78D-43D8-ACA7-71097FE754EE@redivi.com> On Jul 17, 2006, at 4:17 PM, Phillip J. Eby wrote: > At 03:07 PM 7/17/2006 -0700, Bob Ippolito wrote: > >> On Jul 17, 2006, at 2:47 PM, Bob Ippolito wrote: >> >>> >>> On Jul 17, 2006, at 2:33 PM, Phillip J. Eby wrote: >>> >>>> At 12:26 AM 7/17/2006 -0700, Bob Ippolito wrote: >>>>> Is there any reason why setuptools totally ignores extra_path when >>>>> using compatibility mode (e.g. calling .run() manually or -- >>>>> single- >>>>> version-externally-managed)? >>>> >>>> It's probably an unintended side effect of the fact that setuptools >>>> doesn't support extra_path. >>> >>> Why not support it with --single-version-externally-managed? >> >> And further, it should probably issue some kind of warning to let >> people know that setuptools is ignoring this option in any scenario >> when it is ignored... I didn't even notice until I started looking in >> my site-packages folder to uninstall stuff. >> >> In this particular case I don't need --root or --record because I >> already am tracking all of that with bdist_mpkg in a different way. >> >> The reason I want single-version-externally-managed behavior in the >> first place is because I don't want to have to try and deal with >> having the packages try and edit easy_install.pth. I suppose I could >> have it install in an egg-dir and drop in a separate pth, but ideally >> I wouldn't have to do anything. All I want is the old behavior plus >> the egg_info metadata. > > Erm, ok, but the old behavior creates a .pth file, so you're kind > of confusing me here. The other weird bit is that install_egg_info > would need to install to the extra_path directory. Oy. The old behavior creates a .pth file, but I don't have anything to do with it because that was part of the distutils install command. If I were to install as an egg I would have to write the code that creates this new .pth file. > And that's not the worst of it. "install" computes its target > paths during finalize_options(), so there's no way to know at that > point whether run() is going to be called in backward-compatibility > mode. So, there doesn't seem to be a way to implement extra_path > for backward-compatibility mode that doesn't set the --root or -- > single-version-externally-managed options. > > If you can live with that limitation (explicitly setting one or > both of those options when making a super call), I can have it > respect extra_path. But it's a no-go in any other circumstance, > I'm afraid. I don't have any problem explicitly setting either or both of those options. I'm only targeting setuptools, I don't care about old distutils compatibility... I just want something close to the old behavior, for now. -bob From pje at telecommunity.com Tue Jul 18 18:04:04 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Jul 2006 12:04:04 -0400 Subject: [Distutils] setuptools and extra_path In-Reply-To: <75E05796-F78D-43D8-ACA7-71097FE754EE@redivi.com> References: <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> At 04:29 PM 7/17/2006 -0700, Bob Ippolito wrote: >On Jul 17, 2006, at 4:17 PM, Phillip J. Eby wrote: >>If you can live with that limitation (explicitly setting one or >>both of those options when making a super call), I can have it >>respect extra_path. But it's a no-go in any other circumstance, >>I'm afraid. > >I don't have any problem explicitly setting either or both of those >options. I'm only targeting setuptools, I don't care about old >distutils compatibility... I just want something close to the old >behavior, for now. Okay, it's in the 0.7 and 0.6 SVN trunks now; please test (ez_setup.py setuptools=dev06) and let me know if it works for you. From pje at telecommunity.com Tue Jul 18 18:59:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Jul 2006 12:59:11 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <05D15BCF-8880-4DA1-99E0-52FE69EA1DB2@redivi.com> References: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: >That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision). >Any chance of getting this in setuptools 0.6 or should I >start adding MANIFEST.in files to the relevant projects? Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev-r50703. It even handles date and SVN revision tags correctly, by converting them to a single --tag-build string and disabling the other tagging options. So if you just build from an sdist without doing anything special, you get the exact same version the sdist was built with, regardless of how the version was originally specified. From bob at redivi.com Tue Jul 18 19:11:54 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 10:11:54 -0700 Subject: [Distutils] setuptools and extra_path In-Reply-To: <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> References: <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> Message-ID: On Jul 18, 2006, at 9:04 AM, Phillip J. Eby wrote: > At 04:29 PM 7/17/2006 -0700, Bob Ippolito wrote: >> On Jul 17, 2006, at 4:17 PM, Phillip J. Eby wrote: >>> If you can live with that limitation (explicitly setting one or >>> both of those options when making a super call), I can have it >>> respect extra_path. But it's a no-go in any other circumstance, >>> I'm afraid. >> >> I don't have any problem explicitly setting either or both of those >> options. I'm only targeting setuptools, I don't care about old >> distutils compatibility... I just want something close to the old >> behavior, for now. > > Okay, it's in the 0.7 and 0.6 SVN trunks now; please test > (ez_setup.py setuptools=dev06) and let me know if it works for you. seems to work, thanks! -bob From pje at telecommunity.com Tue Jul 18 19:42:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Jul 2006 13:42:32 -0400 Subject: [Distutils] setuptools and extra_path In-Reply-To: References: <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060718134122.03bab458@sparrow.telecommunity.com> At 10:11 AM 7/18/2006 -0700, Bob Ippolito wrote: >On Jul 18, 2006, at 9:04 AM, Phillip J. Eby wrote: > >>At 04:29 PM 7/17/2006 -0700, Bob Ippolito wrote: >>>On Jul 17, 2006, at 4:17 PM, Phillip J. Eby wrote: >>>>If you can live with that limitation (explicitly setting one or >>>>both of those options when making a super call), I can have it >>>>respect extra_path. But it's a no-go in any other circumstance, >>>>I'm afraid. >>> >>>I don't have any problem explicitly setting either or both of those >>>options. I'm only targeting setuptools, I don't care about old >>>distutils compatibility... I just want something close to the old >>>behavior, for now. >> >>Okay, it's in the 0.7 and 0.6 SVN trunks now; please test >>(ez_setup.py setuptools=dev06) and let me know if it works for you. > >seems to work, thanks! > >-bob Great; please check out the sdist/egg_info version thing too. From bob at redivi.com Tue Jul 18 19:57:43 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 10:57:43 -0700 Subject: [Distutils] setuptools and extra_path In-Reply-To: <5.1.1.6.0.20060718134122.03bab458@sparrow.telecommunity.com> References: <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> <5.1.1.6.0.20060718134122.03bab458@sparrow.telecommunity.com> Message-ID: On Jul 18, 2006, at 10:42 AM, Phillip J. Eby wrote: > At 10:11 AM 7/18/2006 -0700, Bob Ippolito wrote: > >> On Jul 18, 2006, at 9:04 AM, Phillip J. Eby wrote: >> >>> At 04:29 PM 7/17/2006 -0700, Bob Ippolito wrote: >>>> On Jul 17, 2006, at 4:17 PM, Phillip J. Eby wrote: >>>>> If you can live with that limitation (explicitly setting one or >>>>> both of those options when making a super call), I can have it >>>>> respect extra_path. But it's a no-go in any other circumstance, >>>>> I'm afraid. >>>> >>>> I don't have any problem explicitly setting either or both of those >>>> options. I'm only targeting setuptools, I don't care about old >>>> distutils compatibility... I just want something close to the old >>>> behavior, for now. >>> >>> Okay, it's in the 0.7 and 0.6 SVN trunks now; please test >>> (ez_setup.py setuptools=dev06) and let me know if it works for you. >> >> seems to work, thanks! >> >> -bob > > Great; please check out the sdist/egg_info version thing too. Looks right, I tried both a release and dev configuration and it preserved the right metadata in setup.cfg. I have not tried a setup.cfg that had any other settings in it though. -bob From pje at telecommunity.com Tue Jul 18 20:32:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Jul 2006 14:32:05 -0400 Subject: [Distutils] setuptools and extra_path In-Reply-To: References: <5.1.1.6.0.20060718134122.03bab458@sparrow.telecommunity.com> <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> <5.1.1.6.0.20060718134122.03bab458@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060718142812.0264c498@sparrow.telecommunity.com> At 10:57 AM 7/18/2006 -0700, Bob Ippolito wrote: >On Jul 18, 2006, at 10:42 AM, Phillip J. Eby wrote: > >>At 10:11 AM 7/18/2006 -0700, Bob Ippolito wrote: >> >>>On Jul 18, 2006, at 9:04 AM, Phillip J. Eby wrote: >>> >>>>At 04:29 PM 7/17/2006 -0700, Bob Ippolito wrote: >>>>>On Jul 17, 2006, at 4:17 PM, Phillip J. Eby wrote: >>>>>>If you can live with that limitation (explicitly setting one or >>>>>>both of those options when making a super call), I can have it >>>>>>respect extra_path. But it's a no-go in any other circumstance, >>>>>>I'm afraid. >>>>> >>>>>I don't have any problem explicitly setting either or both of those >>>>>options. I'm only targeting setuptools, I don't care about old >>>>>distutils compatibility... I just want something close to the old >>>>>behavior, for now. >>>> >>>>Okay, it's in the 0.7 and 0.6 SVN trunks now; please test >>>>(ez_setup.py setuptools=dev06) and let me know if it works for you. >>> >>>seems to work, thanks! >>> >>>-bob >> >>Great; please check out the sdist/egg_info version thing too. > >Looks right, I tried both a release and dev configuration and it >preserved the right metadata in setup.cfg. I have not tried a >setup.cfg that had any other settings in it though. I've tested it with that (by building sdists of setuptools) and it works. The code that's used to edit setup.cfg is the same that's used by "setopt" and "saveopts", so it's had some prior testing for that aspect anyway. I was mainly wanting to make sure that building sdists created this way works properly. Oh, actually, the specific thing I wanted to make sure of was that on platforms with hard linking available it 1) works correctly and 2) doesn't modify the original setup.cfg. The "sdist" command can create an archive tree using hard linking, so I included some code to work around that, but it doesn't execute on Windows so I can't test it in anger, so to speak, and haven't had a moment yet to check it out on a Linux box. If you have a setup.cfg, it should delete the one in the temporary build tree and then recopy it from the source, before editing it. From bob at redivi.com Tue Jul 18 20:39:36 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 11:39:36 -0700 Subject: [Distutils] setuptools and extra_path In-Reply-To: <5.1.1.6.0.20060718142812.0264c498@sparrow.telecommunity.com> References: <5.1.1.6.0.20060718134122.03bab458@sparrow.telecommunity.com> <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717173037.02e915f8@sparrow.telecommunity.com> <500119D8-18A2-4A10-9316-BDAA8156C099@redivi.com> <5.1.1.6.0.20060717191200.038b0198@sparrow.telecommunity.com> <5.1.1.6.0.20060718120313.040f5110@sparrow.telecommunity.com> <5.1.1.6.0.20060718134122.03bab458@sparrow.telecommunity.com> <5.1.1.6.0.20060718142812.0264c498@sparrow.telecommunity.com> Message-ID: <2AA67ED4-23A2-4AF0-A9EB-03BDBF90331B@redivi.com> On Jul 18, 2006, at 11:32 AM, Phillip J. Eby wrote: > At 10:57 AM 7/18/2006 -0700, Bob Ippolito wrote: > >> On Jul 18, 2006, at 10:42 AM, Phillip J. Eby wrote: >> >>> At 10:11 AM 7/18/2006 -0700, Bob Ippolito wrote: >>> >>>> On Jul 18, 2006, at 9:04 AM, Phillip J. Eby wrote: >>>> >>>>> At 04:29 PM 7/17/2006 -0700, Bob Ippolito wrote: >>>>>> On Jul 17, 2006, at 4:17 PM, Phillip J. Eby wrote: >>>>>>> If you can live with that limitation (explicitly setting one or >>>>>>> both of those options when making a super call), I can have it >>>>>>> respect extra_path. But it's a no-go in any other circumstance, >>>>>>> I'm afraid. >>>>>> >>>>>> I don't have any problem explicitly setting either or both of >>>>>> those >>>>>> options. I'm only targeting setuptools, I don't care about old >>>>>> distutils compatibility... I just want something close to the old >>>>>> behavior, for now. >>>>> >>>>> Okay, it's in the 0.7 and 0.6 SVN trunks now; please test >>>>> (ez_setup.py setuptools=dev06) and let me know if it works for >>>>> you. >>>> >>>> seems to work, thanks! >>>> >>>> -bob >>> >>> Great; please check out the sdist/egg_info version thing too. >> >> Looks right, I tried both a release and dev configuration and it >> preserved the right metadata in setup.cfg. I have not tried a >> setup.cfg that had any other settings in it though. > > I've tested it with that (by building sdists of setuptools) and it > works. The code that's used to edit setup.cfg is the same that's > used by "setopt" and "saveopts", so it's had some prior testing for > that aspect anyway. I was mainly wanting to make sure that > building sdists created this way works properly. > > Oh, actually, the specific thing I wanted to make sure of was that > on platforms with hard linking available it 1) works correctly and > 2) doesn't modify the original setup.cfg. The "sdist" command can > create an archive tree using hard linking, so I included some code > to work around that, but it doesn't execute on Windows so I can't > test it in anger, so to speak, and haven't had a moment yet to > check it out on a Linux box. If you have a setup.cfg, it should > delete the one in the temporary build tree and then recopy it from > the source, before editing it. Mac OS X supports hard linking, and I did not see any of my setup.cfg files change while I was testing... so it seems it did what it was supposed to do without any ill side-effects. -bob From pje at telecommunity.com Thu Jul 20 22:57:30 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 20 Jul 2006 16:57:30 -0400 Subject: [Distutils] setuptools 0.6 release candidate 1 Message-ID: <5.1.1.6.0.20060720165523.02622318@sparrow.telecommunity.com> setuptools 0.6c1 has just been released. I'm shooting for a final release of 0.6 in about two weeks, so get those bug reports in pronto. :) From chtaylo3 at vt.edu Thu Jul 20 23:18:47 2006 From: chtaylo3 at vt.edu (chtaylo3) Date: Thu, 20 Jul 2006 17:18:47 -0400 Subject: [Distutils] Import error from Trac for Setuptools Message-ID: <44F0022F@zathras> Hello all, I'm running Tracd as a normal user. When I try and access the tracd server from a remote computer, I get the error mentioned at the end of this post. It looks like a new Workset object is being created with no parameters. So because entries==NONE then entires=sys.path in the constructor. It then calls add_entry which calls find_distributions(). Somehow (I'm not sure how), we get into find_on_path where os.listdir() is called, and this is where it blows up. >From the errors given, it looks like it's trying to do a listdir on the contents of /usr/bin and it get's into permissions trouble. However, /usr/bin is not part of $PYTHONPATH and therefore should not be in the results returned from sys.path. When I run python from the shell, and import sys and then do a sys.path I get the following output: ['', '/usr/lib/python2.4/site-packages/setuptools-0.6b4-py2.4.egg', '/usr/lib/python2.4', '/usr/local/lib/svn-python', '/usr/local/lib/svn-python/svn', '/usr/local/lib/svn-python/libsvn', '/usr/lib/python24.zip', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages'] So I have two questions 1) From a simple import command, how are we getting into find_on_path when it's find_distributions that is being called? 2) What could have interjected the /usr/bin dir into the $PYTHONPATH variable that reuslts in us calling listidr on /usr/bin? Thank you for your help, Christopher Below is the error: Exception happened during processing of request from ('10.11.1.142', 1543) Traceback (most recent call last): File "/usr/lib/python2.4/SocketServer.py", line 463, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.4/SocketServer.py", line 254, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.4/SocketServer.py", line 521, in __init__ self.handle() File "/usr/lib/python2.4/BaseHTTPServer.py", line 314, in handle self.handle_one_request() File "/usr/lib/python2.4/BaseHTTPServer.py", line 308, in handle_one_request method() File "/usr/lib/python2.4/site-packages/trac/web/standalone.py", line 259, in do_GET self._do_trac_req() File "/usr/lib/python2.4/site-packages/trac/web/standalone.py", line 285, in _do_trac_req env = get_environment(req, opts) File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 335, in get_environment return _open_environment(env_path, threaded) File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 51, in _open_environment env_cache[env_path] = open_environment(env_path) File "/usr/lib/python2.4/site-packages/trac/env.py", line 374, in open_environment env = Environment(env_path) File "/usr/lib/python2.4/site-packages/trac/env.py", line 85, in __init__ from trac.loader import load_components File "/usr/lib/python2.4/site-packages/trac/loader.py", line 21, in ? import pkg_resources File "/usr/lib/python2.4/site-packages/setuptools-0.6b4-py2.4.egg/pkg_resources.py" , line 2470, in ? working_set = WorkingSet() File "/usr/lib/python2.4/site-packages/setuptools-0.6b4-py2.4.egg/pkg_resources.py" , line 343, in __init__ self.add_entry(entry) File "/usr/lib/python2.4/site-packages/setuptools-0.6b4-py2.4.egg/pkg_resources.py" , line 358, in add_entry for dist in find_distributions(entry, True): File "/usr/lib/python2.4/site-packages/setuptools-0.6b4-py2.4.egg/pkg_resources.py" , line 1577, in find_on_path for entry in os.listdir(path_item): OSError: [Errno 13] Permission denied: '/usr/bin' From pje at telecommunity.com Fri Jul 21 00:17:04 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 20 Jul 2006 18:17:04 -0400 Subject: [Distutils] Import error from Trac for Setuptools In-Reply-To: <44F0022F@zathras> Message-ID: <5.1.1.6.0.20060720181214.02023798@sparrow.telecommunity.com> At 05:18 PM 7/20/2006 -0400, chtaylo3 wrote: >Hello all, > >I'm running Tracd as a normal user. When I try and access the tracd server >from a remote computer, I get the error mentioned at the end of this >post. It >looks like a new Workset object is being created with no parameters. So >because entries==NONE then entires=sys.path in the constructor. It then >calls >add_entry which calls find_distributions(). Somehow (I'm not sure how), we >get into find_on_path where os.listdir() is called, and this is where it >blows >up. > > >From the errors given, it looks like it's trying to do a listdir on the >contents of /usr/bin and it get's into permissions trouble. However, >/usr/bin >is not part of $PYTHONPATH and therefore should not be in the results >returned >from sys.path. > >When I run python from the shell, and import sys and then do a sys.path I get >the following output: > >['', '/usr/lib/python2.4/site-packages/setuptools-0.6b4-py2.4.egg', >'/usr/lib/python2.4', '/usr/local/lib/svn-python', >'/usr/local/lib/svn-python/svn', '/usr/local/lib/svn-python/libsvn', >'/usr/lib/python24.zip', '/usr/lib/python2.4/plat-linux2', >'/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', >'/usr/lib/python2.4/site-packages'] > >So I have two questions > >1) From a simple import command, how are we getting into find_on_path when >it's find_distributions that is being called? find_distributions returns a dynamically-chosen generator, based on whether the path is a directory or a zipfile (or other type of importer). In this case, it's find_on_path, which looks in a directory to see what projects are present. >2) What could have interjected the /usr/bin dir into the $PYTHONPATH variable >that reuslts in us calling listidr on /usr/bin? It's not being injected in $PYTHONPATH. What's presumably happening is that the script that starts trac is in /usr/bin. Python always includes the startup script's directory in sys.path. Thus, a quick way to work around this would be for you to copy the script from /usr/bin to some other directory, and run that copy instead of the one in /usr/bin. That way, it won't scan a directory you don't have permissions to read. Meanwhile, I'll make sure that the problem (of handling directory permission errors gracefully) is addressed in a future release of setuptools. Thanks for the very detailed and well-investigated problem report! From strawman at astraw.com Fri Jul 21 20:20:23 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 21 Jul 2006 11:20:23 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> References: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> Message-ID: <44C11AE7.3030708@astraw.com> Phillip J. Eby wrote: > At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: > >> That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision). >> Any chance of getting this in setuptools 0.6 or should I >> start adding MANIFEST.in files to the relevant projects? >> > > Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev-r50703. It > even handles date and SVN revision tags correctly, by converting them to a > single --tag-build string and disabling the other tagging options. So if > you just build from an sdist without doing anything special, you get the > exact same version the sdist was built with, regardless of how the version > was originally specified. > I'm glad this is being worked on. But a related issue is still biting me with setuptools 0.6c1 in my stdeb package (which builds debian source packages from unmodified setup.py scripts) : Any distutils commands using "self.distribution.get_version()" still get tagged (at least with the svn revision), even if they're being built from the sdist-generated .tar.gz package. Not knowing the innards of setuptools very well, one idea would be to add something to the .egg-info built by sdist that tells future runs of setuptools not to add tags. This keeps setup.cfg from getting modified but still has the right effect. There's probably a flaw I haven't thought of, though... Cheers! Andrew From strawman at astraw.com Fri Jul 21 20:51:53 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 21 Jul 2006 11:51:53 -0700 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> References: <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> Message-ID: <44C12249.3030100@astraw.com> Phillip J. Eby wrote: > At 08:42 AM 7/4/2006 +0200, Matthias Klose wrote: > >> Phillip J. Eby writes: >> >>>> what about installing the .egg-info directory without version >>>> information when --single-version-externally-managed is used? >>>> >>> Note that the .pyc files will be built for a specific Python version; >>> that's why the version number is there. >>> >> yes, but this shouldn't be necessary if you install into a path which >> is known to belong to a specific python version >> (i.e. /usr/lib/python2.4/site-pacakges) >> > > Setuptools doesn't inspect the path it's installing to in order to know > whether it happens to contain the Python version already. :) I don't see > what problem you're trying to solve here. > The problem he's trying to solve is that the new Debian Python policy allows a single directory tree of .py files (not the .pyc and .pyo files) to serve all installed Python versions. The .pyc files get placed in a separate tree which is, obviously, versioned. I don't see what problem you're trying to solve by having the python version number in the .egg-info/ directory name. :) Is there any Python-version-specific info the in the .egg-info directory? How 'bout in the case when it is installed with --single-version-externally-managed command, which what we're talking about with Debian, anyway? Anyhow, this is also creating an issue for me with stdeb because I'm trying to get the generated source packages as close as possible to Debian policy. Cheers! Andrew From pje at telecommunity.com Fri Jul 21 21:36:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Jul 2006 15:36:47 -0400 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <44C12249.3030100@astraw.com> References: <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> At 11:51 AM 7/21/2006 -0700, Andrew Straw wrote: >I don't see what problem you're trying to solve by having the python >version number in the .egg-info/ directory name. :) Is there any >Python-version-specific info the in the .egg-info directory? Yes. > How 'bout >in the case when it is installed with >--single-version-externally-managed command, which what we're talking >about with Debian, anyway? Yes. >Anyhow, this is also creating an issue for me with stdeb because I'm >trying to get the generated source packages as close as possible to >Debian policy. Unfortunately, Debian's policy -- especially the idea of mixing paths for multiple Python versions -- doesn't mesh well with reality. Python projects don't always install the same files for a given Python version, or the same content in the same files. Setuptools is only a relatively minor example of this; other Python projects do far more customization of the install process (by Python version and dependencies) than setuptools does of its own. Bluntly, mixing Python versions on the same path for *installed* packages (whether installed by the distutils or setuptools) borders on insanity. (More precisely, it sounds like exactly the sort of thing to do if one wanted to create all sorts of problems for which one could then claim credit for fixing as part of a "quality assurance" effort, constantly patching upstream packages to work around the problems, thereby cementing in one's own mind the importance of doing QA work to fix those upstream authors' shoddy work that fails to live up to your distribution's "quality" standards. And yet somehow, nobody will ever seem to notice that it's the policy *itself* that's causing the problems, because of *course* those upstream developers know *nothing* about quality assurance... sigh. But that's a rant for another day.) Anyway, in this case at least, the reason you're running into a problem is because there's an inherent problem here, not because setuptools is doing something it shouldn't be. From pje at telecommunity.com Fri Jul 21 21:40:58 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Jul 2006 15:40:58 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <44C11AE7.3030708@astraw.com> References: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> At 11:20 AM 7/21/2006 -0700, Andrew Straw wrote: >Phillip J. Eby wrote: > > At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: > > > >> That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision). > >> Any chance of getting this in setuptools 0.6 or should I > >> start adding MANIFEST.in files to the relevant projects? > >> > > > > Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev-r50703. It > > even handles date and SVN revision tags correctly, by converting them to a > > single --tag-build string and disabling the other tagging options. So if > > you just build from an sdist without doing anything special, you get the > > exact same version the sdist was built with, regardless of how the version > > was originally specified. > > >I'm glad this is being worked on. But a related issue is still biting me >with setuptools 0.6c1 in my stdeb package (which builds debian source >packages from unmodified setup.py scripts) : > >Any distutils commands using "self.distribution.get_version()" still get >tagged (at least with the svn revision), even if they're being built >from the sdist-generated .tar.gz package. I'm having trouble following your question. If you generate an sdist with an SVN revision in the distribution name, then anything you run on the unpacked sdist should result in the same version number that generated the sdist. So if I build a source distribution for "setuptools-0.6c1dev-r50703.tar.gz", then anything you do with that source distribution should end up with 0.6c1dev-r50703 as the version number. If you have something that *doesn't* end up with that version number, it's a bug. If it *does* have that version number, then it's doing the right thing. The point of this change was to make it so that anything you build from an sdist has the same version number as the sdist was built with. >Not knowing the innards of setuptools very well, one idea would be to >add something to the .egg-info built by sdist that tells future runs of >setuptools not to add tags. This keeps setup.cfg from getting modified >but still has the right effect. There's probably a flaw I haven't >thought of, though... I'm still not understanding what you're getting at; this should be working correctly now in 0.6c1, so if it's not, please give me some steps to reproduce so I can find out what's wrong. Thanks. From tseaver at palladion.com Fri Jul 21 21:56:59 2006 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 21 Jul 2006 15:56:59 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> References: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <44C11AE7.3030708@astraw.com> <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> Message-ID: <44C1318B.6090400@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote: > At 11:20 AM 7/21/2006 -0700, Andrew Straw wrote: >> Phillip J. Eby wrote: >>> At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: >>> >>>> That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision). >>>> Any chance of getting this in setuptools 0.6 or should I >>>> start adding MANIFEST.in files to the relevant projects? >>>> >>> Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev-r50703. It >>> even handles date and SVN revision tags correctly, by converting them to a >>> single --tag-build string and disabling the other tagging options. So if >>> you just build from an sdist without doing anything special, you get the >>> exact same version the sdist was built with, regardless of how the version >>> was originally specified. >>> >> I'm glad this is being worked on. But a related issue is still biting me >> with setuptools 0.6c1 in my stdeb package (which builds debian source >> packages from unmodified setup.py scripts) : >> >> Any distutils commands using "self.distribution.get_version()" still get >> tagged (at least with the svn revision), even if they're being built >>from the sdist-generated .tar.gz package. 'get_version()' returns the Python version, right? Using it makes the egg "version-specific." > I'm having trouble following your question. If you generate an sdist with > an SVN revision in the distribution name, then anything you run on the > unpacked sdist should result in the same version number that generated the > sdist. > > So if I build a source distribution for > "setuptools-0.6c1dev-r50703.tar.gz", then anything you do with that source > distribution should end up with 0.6c1dev-r50703 as the version number. If > you have something that *doesn't* end up with that version number, it's a > bug. If it *does* have that version number, then it's doing the right > thing. The point of this change was to make it so that anything you build > from an sdist has the same version number as the sdist was built with. > > >> Not knowing the innards of setuptools very well, one idea would be to >> add something to the .egg-info built by sdist that tells future runs of >> setuptools not to add tags. This keeps setup.cfg from getting modified >> but still has the right effect. There's probably a flaw I haven't >> thought of, though... > > I'm still not understanding what you're getting at; this should be working > correctly now in 0.6c1, so if it's not, please give me some steps to > reproduce so I can find out what's wrong. Thanks. I think he's talking about the fact that the *python version number" gets embedded in the egg name, rather than the SVN revision ID for the egg. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEwTGL+gerLs4ltQ4RAtaDAJ0XtdK0yatlRm6odgkXWrO4Yo7XjACfb2vb 0eQB1/abwuWjX4SbXBqamgE= =AORa -----END PGP SIGNATURE----- From strawman at astraw.com Fri Jul 21 22:10:47 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 21 Jul 2006 13:10:47 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> References: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> Message-ID: <44C134C7.40705@astraw.com> Phillip J. Eby wrote: > At 11:20 AM 7/21/2006 -0700, Andrew Straw wrote: >> Phillip J. Eby wrote: >> > At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: >> > >> >> That's not a bad idea (update setup.cfg on sdist w/ --no-svn- >> revision). >> >> Any chance of getting this in setuptools 0.6 or should I >> >> start adding MANIFEST.in files to the relevant projects? >> >> >> > >> > Okay, it's in the trunk now as of 0.7a1dev-r50702 and >> 0.6c1dev-r50703. It >> > even handles date and SVN revision tags correctly, by converting >> them to a >> > single --tag-build string and disabling the other tagging options. >> So if >> > you just build from an sdist without doing anything special, you >> get the >> > exact same version the sdist was built with, regardless of how the >> version >> > was originally specified. >> > >> I'm glad this is being worked on. But a related issue is still biting me >> with setuptools 0.6c1 in my stdeb package (which builds debian source >> packages from unmodified setup.py scripts) : >> >> Any distutils commands using "self.distribution.get_version()" still get >> tagged (at least with the svn revision), even if they're being built >> from the sdist-generated .tar.gz package. > > I'm having trouble following your question. If you generate an sdist > with an SVN revision in the distribution name, then anything you run > on the unpacked sdist should result in the same version number that > generated the sdist. > > So if I build a source distribution for > "setuptools-0.6c1dev-r50703.tar.gz", then anything you do with that > source distribution should end up with 0.6c1dev-r50703 as the version > number. If you have something that *doesn't* end up with that version > number, it's a bug. If it *does* have that version number, then it's > doing the right thing. The point of this change was to make it so > that anything you build from an sdist has the same version number as > the sdist was built with. OK, I guess I have an issue with "sdist" build packages made with plain old distutils vs. setuptools. See below. > > >> Not knowing the innards of setuptools very well, one idea would be to >> add something to the .egg-info built by sdist that tells future runs of >> setuptools not to add tags. This keeps setup.cfg from getting modified >> but still has the right effect. There's probably a flaw I haven't >> thought of, though... > > I'm still not understanding what you're getting at; this should be > working correctly now in 0.6c1, so if it's not, please give me some > steps to reproduce so I can find out what's wrong. Thanks. > Steps to reproduce: 1) download and expand matplotlib-0.87.4.tar.gz; cd into that directory. (This was built with plain old distutils sdist.) 2) python -c "import setuptools; execfile('setup.py')" egg_info 3) diff -u PKG-INFO lib/matplotlib.egg-info/PKG-INFO --- PKG-INFO 2006-07-11 19:24:15.000000000 -0700 +++ lib/matplotlib.egg-info/PKG-INFO 2006-07-21 12:49:07.000000000 -0700 @@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: matplotlib -Version: 0.87.4 +Version: 0.87.4.dev-r0 Summary: Matlab(TM) style python plotting package Home-page: http://matplotlib.sourceforge.net Author: John D. Hunter (I'm just using the egg_info command as an example here, but there are other ways to get at Command.distribution.get_version().) Note that building the source distribution with python -c "import setuptools; execfile('setup.py')" sdist DOES result in the same version numbers. (The version number becomes "0.87.4.dev-r0" -- the source has a setup.cfg with "tag_build = .dev" and "tag_svn_revision = 1"). I see now that this could be considered a matplotlib bug -- it has a setup.cfg with setuptools options but distributes packages not built with setuptools. Let me know if you think this is a matplotlib bug. OK, now after some more playing around, I can see that I can generate an sdist using the following (we don't want ".dev-r0" in release name, after all). python -c "import setuptools; execfile('setup.py')" egg_info --tag-build='' --no-svn-revision sdist So is this the recommended sdist-building procedure for packages that ship a setup.cfg with [egg_info] options set? From tseaver at palladion.com Fri Jul 21 21:56:59 2006 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 21 Jul 2006 15:56:59 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> References: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <44C11AE7.3030708@astraw.com> <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> Message-ID: <44C1318B.6090400@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote: > At 11:20 AM 7/21/2006 -0700, Andrew Straw wrote: >> Phillip J. Eby wrote: >>> At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: >>> >>>> That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision). >>>> Any chance of getting this in setuptools 0.6 or should I >>>> start adding MANIFEST.in files to the relevant projects? >>>> >>> Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev-r50703. It >>> even handles date and SVN revision tags correctly, by converting them to a >>> single --tag-build string and disabling the other tagging options. So if >>> you just build from an sdist without doing anything special, you get the >>> exact same version the sdist was built with, regardless of how the version >>> was originally specified. >>> >> I'm glad this is being worked on. But a related issue is still biting me >> with setuptools 0.6c1 in my stdeb package (which builds debian source >> packages from unmodified setup.py scripts) : >> >> Any distutils commands using "self.distribution.get_version()" still get >> tagged (at least with the svn revision), even if they're being built >>from the sdist-generated .tar.gz package. 'get_version()' returns the Python version, right? Using it makes the egg "version-specific." > I'm having trouble following your question. If you generate an sdist with > an SVN revision in the distribution name, then anything you run on the > unpacked sdist should result in the same version number that generated the > sdist. > > So if I build a source distribution for > "setuptools-0.6c1dev-r50703.tar.gz", then anything you do with that source > distribution should end up with 0.6c1dev-r50703 as the version number. If > you have something that *doesn't* end up with that version number, it's a > bug. If it *does* have that version number, then it's doing the right > thing. The point of this change was to make it so that anything you build > from an sdist has the same version number as the sdist was built with. > > >> Not knowing the innards of setuptools very well, one idea would be to >> add something to the .egg-info built by sdist that tells future runs of >> setuptools not to add tags. This keeps setup.cfg from getting modified >> but still has the right effect. There's probably a flaw I haven't >> thought of, though... > > I'm still not understanding what you're getting at; this should be working > correctly now in 0.6c1, so if it's not, please give me some steps to > reproduce so I can find out what's wrong. Thanks. I think he's talking about the fact that the *python version number" gets embedded in the egg name, rather than the SVN revision ID for the egg. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEwTGL+gerLs4ltQ4RAtaDAJ0XtdK0yatlRm6odgkXWrO4Yo7XjACfb2vb 0eQB1/abwuWjX4SbXBqamgE= =AORa -----END PGP SIGNATURE----- From strawman at astraw.com Fri Jul 21 22:32:33 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 21 Jul 2006 13:32:33 -0700 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> References: <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> Message-ID: <44C139E1.4060106@astraw.com> Phillip, thanks for your patient attempts to make this all clear. I appreciate it. Some more comments at the end... Phillip J. Eby wrote: > At 11:51 AM 7/21/2006 -0700, Andrew Straw wrote: >> I don't see what problem you're trying to solve by having the python >> version number in the .egg-info/ directory name. :) Is there any >> Python-version-specific info the in the .egg-info directory? > > Yes. > >> How 'bout >> in the case when it is installed with >> --single-version-externally-managed command, which what we're talking >> about with Debian, anyway? > > Yes. > > >> Anyhow, this is also creating an issue for me with stdeb because I'm >> trying to get the generated source packages as close as possible to >> Debian policy. > > Unfortunately, Debian's policy -- especially the idea of mixing paths > for multiple Python versions -- doesn't mesh well with reality. > Python projects don't always install the same files for a given Python > version, or the same content in the same files. Setuptools is only a > relatively minor example of this; other Python projects do far more > customization of the install process (by Python version and > dependencies) than setuptools does of its own. Bluntly, mixing Python > versions on the same path for *installed* packages (whether installed > by the distutils or setuptools) borders on insanity. > > (More precisely, it sounds like exactly the sort of thing to do if one > wanted to create all sorts of problems for which one could then claim > credit for fixing as part of a "quality assurance" effort, constantly > patching upstream packages to work around the problems, thereby > cementing in one's own mind the importance of doing QA work to fix > those upstream authors' shoddy work that fails to live up to your > distribution's "quality" standards. And yet somehow, nobody will ever > seem to notice that it's the policy *itself* that's causing the > problems, because of *course* those upstream developers know > *nothing* about quality assurance... sigh. But that's a rant for > another day.) > > Anyway, in this case at least, the reason you're running into a > problem is because there's an inherent problem here, not because > setuptools is doing something it shouldn't be. > OK, I understand your point. Upon re-reading the new Debian python policy, I see that the idea of having .py files installed in one tree for multiple python versions is optional (albeit "strongly encouraged" as stated at http://www.debian.org/doc/packaging-manuals/python-policy/ap-packaging_tools.html). So, I can personally continue to ignore this supposedly optional aspect of the new Debian Python policy, but I don't know for how long. For example, recent discussions on the debian-python list suggest to me that the use of this new policy is very strongly encouraged rather than merely "optional". Somebody please enlighten me if I'm wrong. I've said it before and I'll say it again -- I'm no Debian (Python) policy expert. So, to ask your advice, then, how would _you_ suggest stdeb install Python packages using the Debian system? Plain old "--single-version-externally-managed" without any .egg-info directory renaming? That would be simple enough for me... From pje at telecommunity.com Fri Jul 21 22:33:17 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Jul 2006 16:33:17 -0400 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <44C134C7.40705@astraw.com> References: <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060721162408.02c36420@sparrow.telecommunity.com> At 01:10 PM 7/21/2006 -0700, Andrew Straw wrote: >1) download and expand matplotlib-0.87.4.tar.gz; cd into that directory. >(This was built with plain old distutils sdist.) Okay, that's where the problem lies, then. If you use distutils sdist, you get what the distutils sdist does. :) If you build an sdist using setuptools (0.6c1), then that sdist will have the same version number as the original. If you build an sdist using distutils OR a version of setuptools less than 0.6c1, you will get... something else. :) >Note that building the source distribution with python -c "import >setuptools; execfile('setup.py')" sdist DOES result in the same version >numbers. (The version number becomes "0.87.4.dev-r0" -- the source has a >setup.cfg with "tag_build = .dev" and "tag_svn_revision = 1"). I see >now that this could be considered a matplotlib bug -- it has a setup.cfg >with setuptools options but distributes packages not built with >setuptools. Let me know if you think this is a matplotlib bug. Sort of. It's more a question of what behavior they want. I think they are trying to walk a fine line of being able to use setuptools without requiring users to have it. >OK, now after some more playing around, I can see that I can generate an >sdist using the following (we don't want ".dev-r0" in release name, >after all). > >python -c "import setuptools; execfile('setup.py')" egg_info >--tag-build='' --no-svn-revision sdist > >So is this the recommended sdist-building procedure for packages that >ship a setup.cfg with [egg_info] options set? Not unless the package creator requests it. Otherwise, you could be stripping version information that's *supposed* to be there. This is why setuptools now ships an automatically modified setup.cfg when it builds an sdist, so that what you get by default is whatever the builder of the sdist wanted. So, in the current situation with matplotlib, one of two things needs to happen: * They need to build their sdist with setuptools, OR * They need to strip the egg_info settings from setup.cfg before building an untagged release with the distutils Either way should result in an sdist that works correctly for setuptools, but it may be that they have to take the latter option in order to avoid requiring setuptools. (I'm not sure whether that's the case or not.) In any case, until 0.6c1, the latter option was the only *possible* option, and it's discussed at some length in the setuptools manual, including alternative ways to manage the process with Subversion. From doko at cs.tu-berlin.de Fri Jul 21 22:46:59 2006 From: doko at cs.tu-berlin.de (Matthias Klose) Date: Fri, 21 Jul 2006 22:46:59 +0200 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> References: <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> Message-ID: <17601.15683.346287.274254@gargle.gargle.HOWL> Phillip J. Eby writes: > Unfortunately, Debian's policy -- especially the idea of mixing paths for > multiple Python versions -- doesn't mesh well with reality. that's a misunderstading, we don't do that. common files are shipped in one location outside of sys.path and linked to a version specific location inside sys.path for every python version. > Python > projects don't always install the same files for a given Python version, or > the same content in the same files. Setuptools is only a relatively minor > example of this; other Python projects do far more customization of the > install process (by Python version and dependencies) than setuptools does > of its own. Bluntly, mixing Python versions on the same path for > *installed* packages (whether installed by the distutils or setuptools) > borders on insanity. again, they are not mixed. I didn't see that many packages with different files between Python versions. could you give some examples? > (More precisely, it sounds like exactly the sort of thing to do if one > wanted to create all sorts of problems for which one could then claim > credit for fixing as part of a "quality assurance" effort, constantly > patching upstream packages to work around the problems, thereby cementing > in one's own mind the importance of doing QA work to fix those upstream > authors' shoddy work that fails to live up to your distribution's "quality" > standards. And yet somehow, nobody will ever seem to notice that it's the > policy *itself* that's causing the problems, because of *course* those > upstream developers know *nothing* about quality assurance... sigh. But > that's a rant for another day.) I don't see the problem you describe. But anyway, doing QA work in Debian for python packages (related to the pakage itself) is - to move around files into FHS conforming locations in the file system (in contrast to the Windows centric view of many upstream modules / packages installing a thing in _one_ location) and keep the package working. FHS is not just a quality standard for Debian. - remove included third party modules and depend on the modules instead which are shipped in the distribution (removed three copies of pytz in the last months). Maybe it's the desire of upstream authors to provide a complete one-click-setup lacking a decent package management system for other platforms, but it should be an option, not the default. - remove files which do not conform to the Debian Free Software Guidelines (DFSG), which would disallow the distribution of an upstream package in Debian. - Fix window'isms like linking extension modules with libpython, convert file encodings, etc. - add exact meta information (dependencies) that allows upgrades, even partial ones, between distribution releases. It would be nice to use upstream meta information, without relying on a newly invented dependency tracking system which doesn't integrate well with existing packaging systems. I think the PEP trying to get this information into the meta-data was rejected as over-specified. - split an upstream package into more than one binary package to avoid unnecessary installations (i.e. install the python-imaging parts, which do not depend on X or Sane). The policy addresses other (Debian related) things like not introducing new debian package names for new python versions (because that requires interaction by FTP adminstrators) and avoiding uploads for every python package when the python version changes and hindering the release process for the distribution (forcing big-bang updates between the unstable and the testing release). So yes, I don't expect upstream authors to know anything about Debian QA and to adjust the packaging instead as a package maintainer. Non-Debian specific changes are usually forwarded to the upstream authors. Upstreams aren't a homogenous group, from some authors I get immediate feedback if I modify something in the package, others don't see a a need to fix a "python setup.py clean" removing half of the distributed files which cannot be regenerated. Matthias From bob at redivi.com Fri Jul 21 22:48:43 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 21 Jul 2006 13:48:43 -0700 Subject: [Distutils] setuptools 0.6b4 released In-Reply-To: <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> References: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com> <5.1.1.6.0.20060721153700.037f3908@sparrow.telecommunity.com> Message-ID: On Jul 21, 2006, at 12:40 PM, Phillip J. Eby wrote: > At 11:20 AM 7/21/2006 -0700, Andrew Straw wrote: >> Phillip J. Eby wrote: >>> At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote: >>> >>>> That's not a bad idea (update setup.cfg on sdist w/ --no-svn- >>>> revision). >>>> Any chance of getting this in setuptools 0.6 or should I >>>> start adding MANIFEST.in files to the relevant projects? >>>> >>> >>> Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev- >>> r50703. It >>> even handles date and SVN revision tags correctly, by converting >>> them to a >>> single --tag-build string and disabling the other tagging >>> options. So if >>> you just build from an sdist without doing anything special, you >>> get the >>> exact same version the sdist was built with, regardless of how >>> the version >>> was originally specified. >>> >> I'm glad this is being worked on. But a related issue is still >> biting me >> with setuptools 0.6c1 in my stdeb package (which builds debian source >> packages from unmodified setup.py scripts) : >> >> Any distutils commands using "self.distribution.get_version()" >> still get >> tagged (at least with the svn revision), even if they're being built >> from the sdist-generated .tar.gz package. > > I'm having trouble following your question. If you generate an > sdist with > an SVN revision in the distribution name, then anything you run on the > unpacked sdist should result in the same version number that > generated the > sdist. > > So if I build a source distribution for > "setuptools-0.6c1dev-r50703.tar.gz", then anything you do with that > source > distribution should end up with 0.6c1dev-r50703 as the version > number. If > you have something that *doesn't* end up with that version number, > it's a > bug. If it *does* have that version number, then it's doing the right > thing. The point of this change was to make it so that anything > you build > from an sdist has the same version number as the sdist was built with. > > >> Not knowing the innards of setuptools very well, one idea would be to >> add something to the .egg-info built by sdist that tells future >> runs of >> setuptools not to add tags. This keeps setup.cfg from getting >> modified >> but still has the right effect. There's probably a flaw I haven't >> thought of, though... > > I'm still not understanding what you're getting at; this should be > working > correctly now in 0.6c1, so if it's not, please give me some steps to > reproduce so I can find out what's wrong. Thanks. It sounds like Andrew just forgot to use --no-svn-revision... -bob From pje at telecommunity.com Fri Jul 21 23:02:02 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Jul 2006 17:02:02 -0400 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <44C139E1.4060106@astraw.com> References: <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060721164341.03e6ec18@sparrow.telecommunity.com> At 01:32 PM 7/21/2006 -0700, Andrew Straw wrote: >Upon re-reading the new Debian python policy, I see that the idea of >having .py files installed in one tree for multiple python versions is >optional (albeit "strongly encouraged" as stated at >http://www.debian.org/doc/packaging-manuals/python-policy/ap-packaging_tools.html). >So, I can personally continue to ignore this supposedly optional aspect >of the new Debian Python policy, but I don't know for how long. For >example, recent discussions on the debian-python list suggest to me that >the use of this new policy is very strongly encouraged rather than >merely "optional". I read the entire policy you linked to, and I don't actually see many problems. It seems to me that the single largest problem in that policy is that it clearly predates the existence of the distutils. It has no conception of a Python *project* or *distribution*, only modules and packages. It's therefore not surprising that it also doesn't encompass such issues as distribution metadata, package data, namespace packages, and the like. It also explains why the policy is so out-of-sync with e.g. PyPI. (I hesitate to see what would happen if somebody tries to package any of my Python projects such as SymbolType or ProxyTypes for Debian: they all are modules in the 'peak' package, but each is distributed as a separate project!) From the extremely limited conceptual position that only modules and packages exist, it's also impossible to understand how you can have different versions of things for different versions of Python, because everything is defined in terms of the modules! However, project distributions are merely *composed* of modules, and what modules get installed (or what distribution metadata is generated) may depend on the Python version. The same project may also have different dependencies, depending on the Python version. These concepts can't be well-understood from the perspective that only modules and packages exist, so until the policy's conceptual underpinning is expanded, it's going to continue to be difficult to squeeze square pegs into the policy's round holes. >So, to ask your advice, then, how would _you_ suggest stdeb install >Python packages using the Debian system? Plain old >"--single-version-externally-managed" without any .egg-info directory >renaming? That would be simple enough for me... That is what I would recommend, yes. Installed, of course, to the standard Python directories, rather than to a shared one, since that would lead to conflicts when alternate versions exist. I don't know enough about Debian to know what it does with package naming, but I do know that e.g. NetBSD package naming conventions include the Python version for which a project is packaged, such that you could have py22-foo, py23-foo, and py24-foo of the same project "foo". IMO, any other way of doing it is an accident waiting to happen... or else a "job security" bonus for the people who write the tools and fix the problems. I guess it's always better to look at it from the bright side. ;) From pje at telecommunity.com Fri Jul 21 23:31:27 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Jul 2006 17:31:27 -0400 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <17601.15683.346287.274254@gargle.gargle.HOWL> References: <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060721170306.03e7f508@sparrow.telecommunity.com> At 10:46 PM 7/21/2006 +0200, Matthias Klose wrote: >again, they are not mixed. I didn't see that many packages with >different files between Python versions. could you give some examples? The one that's most obvious to me is of course setuptools itself; its installation metadata changes between Python versions, to reflect differences in the distutils between Python versions. The most common case of a project shipping different modules are projects that depend on features that were part of the stdlib in a newer Python version, but must be bundled for an older Python version. e.g., the logging module, optparse/optik, doctest, the threading.local facility, etc. I seem to recall Zope releases doing this a lot. >- remove included third party modules and depend on the modules > instead which are shipped in the distribution (removed three copies > of pytz in the last months). Maybe it's the desire of upstream authors > to provide a complete one-click-setup lacking a decent package > management system for other platforms, but it should be an option, > not the default. Luckily, setuptools permits improvement in this area, and certainly I agree that the dependencies *should* be made separate. That's not always possible, however, if you are dealing with the existing base of Python projects. >- add exact meta information (dependencies) that allows upgrades, even > partial ones, between distribution releases. It would be nice to use > upstream meta information, without relying on a newly invented > dependency tracking system which doesn't integrate well with > existing packaging systems. I think the PEP trying to get this > information into the meta-data was rejected as over-specified. Actually, it was only over-specified on *syntax*. The *semantics* on the other hand, are badly *under*-specified. The version syntax was so narrowly-defined that *Python's own* version numbers wouldn't be considered conformant, but the meaning of the names was left entirely unspecified! I could say I wanted "Foo>1.2.7", but there was no definition of what "Foo" would mean. Regarding the rest of your comments (e.g. FHS), I think it would be counterproductive for me to respond in detail, because I do understand that these things that are meaningful within the Debian world-view. However, they are self-fulfilling prophecies in the sense that these forms of "QA" works primarily to ensure the need for more of the same "QA". If this is what is valued, then by all means, feel free to go for as much of it as you like. :) Where this leads to friction with outside work (like Andrew's stdeb) is that the Debian viewpoint doesn't encompass the idea that the proper division of Python modules is according to the projects that provide them. This is an impedance mismatch that means that any tool designed to help get Python projects into Debian (like easy_deb or stdeb or even bdist_deb) is never going to be both reasonably automated *and* reasonably policy-compliant, because the conceptual framework on which the policy is based cannot encompass the idea of projects. It seems that to Debian, the only software project is Debian itself, and anything else is merely a source of components to be broken down and reassembled in Debian's own image. This may or may not be what you want, of course, and I certainly don't object to people doing what they want. :) I'm bringing this up, however, in the hopes of saving more people from slaving away trying to build Python packaging tools for Debian based on a misconception that their work will ever be accepted (or even understood) in the scope of Debian's Python policy, which effectively precludes any useful mapping from PyPI to Debian. From doko at cs.tu-berlin.de Sat Jul 22 00:08:24 2006 From: doko at cs.tu-berlin.de (Matthias Klose) Date: Sat, 22 Jul 2006 00:08:24 +0200 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <5.1.1.6.0.20060721164341.03e6ec18@sparrow.telecommunity.com> References: <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060721164341.03e6ec18@sparrow.telecommunity.com> Message-ID: <17601.20568.542953.211509@gargle.gargle.HOWL> Phillip J. Eby writes: > I read the entire policy you linked to, and I don't actually see many problems. > > It seems to me that the single largest problem in that policy is that it > clearly predates the existence of the distutils. It has no conception of a > Python *project* or *distribution*, only modules and packages. It's > therefore not surprising that it also doesn't encompass such issues as > distribution metadata, package data, namespace packages, and the like. It > also explains why the policy is so out-of-sync with e.g. PyPI. (I hesitate > to see what would happen if somebody tries to package any of my Python > projects such as SymbolType or ProxyTypes for Debian: they all are modules > in the 'peak' package, but each is distributed as a separate project!) The Python policy is just a sub-part of the Debian Policy [1]; the Debian Policy predates PyPi. You are missing the existing bits about i.e. distribution metadata, distributions, etc. [1] http://www.us.debian.org/doc/debian-policy/ I cannot find the term "project" in the distutils documentation. Any pointers? So yes, if peak is a rather complex setup, it might be worthful to have it as an example for a Debian package and to identify omissions in Debian packaging practices and distutils/setuptools. > From the extremely limited conceptual position that only modules and > packages exist, it's also impossible to understand how you can have > different versions of things for different versions of Python, because > everything is defined in terms of the modules! However, project > distributions are merely *composed* of modules, and what modules get > installed (or what distribution metadata is generated) may depend on the > Python version. The same project may also have different dependencies, > depending on the Python version. It is not the goal for Debian to ship every known Python version; we try to ship a minimal count of Python versions, which is current and works for most applications. We discourage shipping deliberate versions of python modules/packages. IIRC, for the past release of Debian sarge, we had two versions of pygtk and two versions of wxwidgets. So it seems that it's quiet possible to do that. I'm not sure what you mean by the generation of distribution metadata and different dependencies. > These concepts can't be well-understood from the perspective that only > modules and packages exist, so until the policy's conceptual underpinning > is expanded, it's going to continue to be difficult to squeeze square pegs > into the policy's round holes. agreed, but it cannot be as open as the possibilities of distutils/setuptools are. Python packages (in the Debian sense) still have to follow [1] and general decisions made by release management. > >So, to ask your advice, then, how would _you_ suggest stdeb install > >Python packages using the Debian system? Plain old > >"--single-version-externally-managed" without any .egg-info directory > >renaming? That would be simple enough for me... > > That is what I would recommend, yes. Installed, of course, to the standard > Python directories, rather than to a shared one, since that would lead to > conflicts when alternate versions exist. > > I don't know enough about Debian to know what it does with package naming, > but I do know that e.g. NetBSD package naming conventions include the > Python version for which a project is packaged, such that you could have > py22-foo, py23-foo, and py24-foo of the same project "foo". That naming scheme was used until recently. As already said, it had several disadvantages for Debian: - manual interaction for each new package name entering the Debian archive - big bang transitions between unstable and testing releases. - higher space requirements for ftp archives, ftp mirrors, local installations, more band with usage - old packages left after package upgrades > IMO, any other > way of doing it is an accident waiting to happen... or else a "job > security" bonus for the people who write the tools and fix the problems. I > guess it's always better to look at it from the bright side. ;) Well, let's see what will happen and lets count the accidents then ;) Many problems that PyPI and setuptools try to solve are well addressed by existing packaging tools for Linux and *BSD distributions. It would be nice to see setuptools to use this infrastructure where available. Linux and *BSD distributions do integrate software well, PyPI and setuptools might do that job better for Python things, but just for Python things. I'm sure you can improve things from both sides. Matthias From bob at redivi.com Sat Jul 22 00:29:47 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 21 Jul 2006 15:29:47 -0700 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <17601.20568.542953.211509@gargle.gargle.HOWL> References: <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060721164341.03e6ec18@sparrow.telecommunity.com> <17601.20568.542953.211509@gargle.gargle.HOWL> Message-ID: On Jul 21, 2006, at 3:08 PM, Matthias Klose wrote: > Phillip J. Eby writes: >> I read the entire policy you linked to, and I don't actually see >> many problems. >> >> It seems to me that the single largest problem in that policy is >> that it >> clearly predates the existence of the distutils. It has no >> conception of a >> Python *project* or *distribution*, only modules and packages. It's >> therefore not surprising that it also doesn't encompass such >> issues as >> distribution metadata, package data, namespace packages, and the >> like. It >> also explains why the policy is so out-of-sync with e.g. PyPI. (I >> hesitate >> to see what would happen if somebody tries to package any of my >> Python >> projects such as SymbolType or ProxyTypes for Debian: they all are >> modules >> in the 'peak' package, but each is distributed as a separate >> project!) > > The Python policy is just a sub-part of the Debian Policy [1]; the > Debian Policy predates PyPi. You are missing the existing bits about > i.e. distribution metadata, distributions, etc. > [1] http://www.us.debian.org/doc/debian-policy/ > > I cannot find the term "project" in the distutils documentation. > Any pointers? > > So yes, if peak is a rather complex setup, it might be worthful to > have it as an example for a Debian package and to identify omissions > in Debian packaging practices and distutils/setuptools. What pje means by "project" is the Distribution class in distutils. Effectively, the group of stuff addressed by a single "setup.py" script is a project. >> IMO, any other >> way of doing it is an accident waiting to happen... or else a "job >> security" bonus for the people who write the tools and fix the >> problems. I >> guess it's always better to look at it from the bright side. ;) > > Well, let's see what will happen and lets count the accidents then ;) > Many problems that PyPI and setuptools try to solve are well addressed > by existing packaging tools for Linux and *BSD distributions. It would > be nice to see setuptools to use this infrastructure where available. > Linux and *BSD distributions do integrate software well, PyPI and > setuptools might do that job better for Python things, but just for > Python things. I'm sure you can improve things from both sides. It doesn't really make any sense for setuptools to use "infrastructure where available" [1]. The metadata it needs must be available with the source code, and it needs to be accessible in a platform agnostic way post-installation. Tools like stdeb are being written so that it's easier to build packages for a given existing infrastructure (inheriting the setuptools metadata "for free"). [1] You could argue that easy_install (and the dependency resolution that happens with the "*_requires" lists) should integrate with existing package installation infrastructure where available, but that's a different discussion entirely. We're talking about assembling packages, not acquisition and installation. -bob From pje at telecommunity.com Sat Jul 22 02:15:28 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Jul 2006 20:15:28 -0400 Subject: [Distutils] python version information in .egg-info directory name In-Reply-To: <17601.20568.542953.211509@gargle.gargle.HOWL> References: <5.1.1.6.0.20060721164341.03e6ec18@sparrow.telecommunity.com> <5.1.1.6.0.20060721151600.037e2b38@sparrow.telecommunity.com> <5.1.1.6.0.20060704100338.01f22880@sparrow.telecommunity.com> <5.1.1.6.0.20060703130254.01f16878@sparrow.telecommunity.com> <5.1.1.6.0.20060609171516.033a4a00@mail.telecommunity.com> <5.1.1.6.0.20060609161725.032fcfc0@mail.telecommunity.com> <17545.53572.694483.215323@gargle.gargle.HOWL> <5.1.1.6.0.20060721164341.03e6ec18@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060721185958.026224a8@sparrow.telecommunity.com> At 12:08 AM 7/22/2006 +0200, Matthias Klose wrote: >Phillip J. Eby writes: > > I read the entire policy you linked to, and I don't actually see many > problems. > > > > It seems to me that the single largest problem in that policy is that it > > clearly predates the existence of the distutils. It has no conception > of a > > Python *project* or *distribution*, only modules and packages. It's > > therefore not surprising that it also doesn't encompass such issues as > > distribution metadata, package data, namespace packages, and the like. It > > also explains why the policy is so out-of-sync with e.g. PyPI. (I > hesitate > > to see what would happen if somebody tries to package any of my Python > > projects such as SymbolType or ProxyTypes for Debian: they all are modules > > in the 'peak' package, but each is distributed as a separate project!) > >The Python policy is just a sub-part of the Debian Policy [1]; the >Debian Policy predates PyPi. You are missing the existing bits about >i.e. distribution metadata, distributions, etc. >[1] http://www.us.debian.org/doc/debian-policy/ I'm referring above to Python distribution metadata, not Debian's. That is, the distribution of a Python project, not a Linux distribution. >I cannot find the term "project" in the distutils documentation. Any pointers? I use the term "project" to refer to the logical thing of which a distutils "distribution" is a physical manifestation. The distutils documentation confusingly uses the word "package" to refer both to what I'm calling a "project", and the notion of an individual Python package. You can tell this by close inspection of the distutils documentation, if you notice that there are many places where the configuration of a "package" (meaning #1) in fact can list multiple "packages" (meaning #2) for inclusion. (Many Python developers have previously commented on this naming ambiguity in the distutils.) I thus prefer to use (and promote the use of) the word "project" for meaning 1, in order to have better communication about what is actually going on. It is intuitive and does not confuse two different notions of "package". >So yes, if peak is a rather complex setup, it might be worthful to >have it as an example for a Debian package and to identify omissions >in Debian packaging practices and distutils/setuptools. This very statement helps to illustrate the impedance mismatch and communication difficulty. You appear to be interpreting my statement that e.g. SymbolTypes and ProxyTypes contain modules in the "peak" package (meaning #2) to imply that peak is or should be a Debian package (meaning #1, or perhaps a new meaning #3!). But this would be the same as concluding that the various Java projects whose packages are contained in the org.apache.* namespace are a single "package" in this way, and should thus be combined into a Debian "org-apache" package -- split out of their respective jars and forcibly recombined! But as with the org.apache.* prefix, the peak.* prefix is merely a namespace that indicates an affiliation and prevents unintentional name conflicts. As the number of distributed Python projects increases (1400+ on PyPI of late), this kind of namespace management will become increasingly important. This is where the term "namespace package" comes from; it was coined several years ago by Zope to distinguish package-as-unit from package-as-namespace. The latter kind of package is still relatively uncommon, since there are relatively few large organizations distributing large projects as split distributions. Unbundling these large projects into smaller pieces is an increasing trend, however, as it allows smaller units to be spun off as sub-projects, each with its own release management and version lifecycle. PEAK and Zope as monolithic projects might contain some elements in alpha or beta releases that, considered in their own right, might be worthy of 1.x or 2.x stable version labelling. Lumping these in together with other components doesn't help anybody, but spinning them off as separate projects allows them to be reused. So far, I've spun off seven such packages from the monolithic PEAK distribution, and there will be more over time. These other packages live in the peak.* namespace, and the monolithic distribution depends on them, but it would not make sense to aggregate them all as one Debian package, since other packages may depend directly on them. SymbolTypes, ProxyTypes, and DecoratorTools are all likely to get used in other projects that would depend on them directly, but not necessarily require any other part of the peak.* namespace. And this, you see, is why I say that the Debian Python policy is based on a limited conceptual framework that doesn't mesh well with a distutils- or setuptools-based world. Mapping "1 Debian package = 1 Python package = 1 project" is inaccurate, because one project may contain multiple Python packages, and a single Python package can be spread across multiple projects. And it's not just PEAK and Zope doing it -- I discovered last year that there's an ll.* namespace package out there that uses an interesting quirk of the distutils installation system to implement a namespace package. It might actually predate Zope's coining of the term "namespace package" for all I know. >I'm not sure what you mean by the generation of distribution metadata >and different dependencies. The PKG-INFO format changed in some Python versions. The entry points that setuptools offers as commands depends on what Python version it's installed with. The dependencies of some projects depend on whether a needed thing is now bundled in Python. For example, there is a standalone "ctypes" project, that is bundled in Python 2.5. A project being installed for 2.5 would have no reason to declare a dependency on ctypes, and it would be entirely reasonable for a setup.py to contain something like: deps = [] if sys.version[:3]<"2.5": deps.append("ctypes>=0.9.6") ... setup( ... install_requires = deps, ... ) Similar code might decide to build alternate extensions, etc. The monolithic PEAK distribution for example includes its own version of the Python expat module, and builds it to replace Python's if it's being installed for Python 2.3 (whose expat interface doesn't include access to the current line number during regular parsing callbacks). (Note: it builds this backported extension as peak.util.pyexpat; it doesn't override the stdlib-supplied pyexpat!) Anyway, now that setuptools exists, the right way for me to have handled that would be to create a separate project, let's say "pyexpat-backport" that provides the 2.4 expat interface for Python 2.3, and then declare that as a dependency if I'm installing under Python 2.3 -- a Python-version-conditional dependency. > > These concepts can't be well-understood from the perspective that only > > modules and packages exist, so until the policy's conceptual underpinning > > is expanded, it's going to continue to be difficult to squeeze square pegs > > into the policy's round holes. > >agreed, but it cannot be as open as the possibilities of >distutils/setuptools are. Python packages (in the Debian sense) still >have to follow [1] and general decisions made by release management. You'll have to clue me in as to which meaning of "package" you're using here. I personally try to use the following terms to be unambiguous: 1. "Project" - a thing that somebody distributes 2. "Python package" - something you can actually import! 3. "System package" - something that is installed with a system packaging tool, like a .rpm 4. "Distribution" - an embodiment of a particular release of a project As far as I can tell, Debian terminology conflates some of these terms. And so long as its vocabulary is thus restricted, there will be an impedance mismatch at the interface where people try to create tools to support mapping #1 and #4 on to Debian's #3. >Many problems that PyPI and setuptools try to solve are well addressed >by existing packaging tools for Linux and *BSD distributions. A similarity in solutions is not the same as similarity in problems. The goals of a system packaging tool and the goals of setuptools are quite different, and in some cases may actually be opposed. :) Setuptools' fundamental goal is to encourage reuse by lowering the transaction cost of depending on another developer's software. Not merely in the sense of lowering *distribution* or *installation* cost, but also enhancing the extensibility and interoperability of the projects themselves. Metadata and entry points facilitate creating *platforms* in Python, such as the joint TurboGears-CherryPy template plugin API. That API couldn't exist without something like setuptools; system packaging tools simply don't play in that space. Now, furthering setuptools' goals *does* require distribution and dependency management... but its "low transaction cost" goal means that it requires a *common* nomenclature for referencing projects. A nomenclature that varied from one packaging system to another would not lower transaction cost, since it would force a developer to learn the ever-changing and mutually incompatible naming conventions of every Linux and BSD variant. The only universal nomenclature available, therefore, was project names. The distutils built distributions using project names, and PyPI displayed project names. Hence, it was and is the right choice for Python to identify projects by those names. Distributions, however, that insist on deconstructing Python projects and creating nomenclature with no mapping to PyPI project names, simply create a policy barrier between those upstream projects and ready access by their users. It increases the transaction cost for providing software to Debian users -- and Debian of course ends up bearing those costs. The efforts of people like Andrew and Vincenzo to create tools that map PyPI projects into Debian packages are therefore in vain; Debian doesn't want to decrease transaction cost, which then leaves the tool developers confused, since their goal is to further reduce transaction costs. I myself was initially baffled by this resistance from Debian representatives, but now I simply accept it as a fact that Debian's goals differ from mine. I do think it's unfortunate, though, because other people seem to keep thinking that they will be able to write a conversion tool and solve a sociopolitical/conceptual problem with a technical solution. It just ain't gonna happen. :) (I don't mean it's unfortunate that Debian has different goals, I just mean it's unfortunate that this fact isn't immediately obvious to the people who keep beating their heads on this particular wall. You can't work to lower the impedance between PyPI and Debian, and still please Debian policy, because the policy itself is the source of the impedance.) >It would be nice to see setuptools to use this infrastructure where available. The --single-version-externally-managed option exists so that setuptools can get out of system packaging tools' way. There's also extensive work that I did to make namespace packages play well with system packaging tools that don't allow more than one system package to provide the same file, although this required what some Pythoneers would consider a horrific abuse of Python's .pth file system. These things were done because people doing work for Debian asked for them, and if anybody asks nicely for other things that I can provide, I'll certainly do so. However, some things just aren't doable. I can't, for example, turn back the clock seven years and make the distutils go away, or even four years to make namespace packages go away, just because Debian policy doesn't grok those concepts yet, or refuses to acknowledge their validity. Even if I agreed with Debian on these points (and I don't), the Python community voted with its feet years ago, and Guido blessed all of them. The distutils were blessed for the stdlib in what, Python 2.1? Namespace packages were blessed for 2.3 (see the "pkgutil" module docs, although they use the term "logical package"). (Guido himself wrote that module, if I recall correctly.) Support for package data (data files found inside package directories) was added in 2.4, and .egg-info distribution metadata was blessed for 2.5. From the Python POV, most of this stuff is ancient history by now. From jorge.vargas at gmail.com Tue Jul 25 05:27:58 2006 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Mon, 24 Jul 2006 23:27:58 -0400 Subject: [Distutils] a patch for distutils + gcc + linux, that was never apply, yet will solve many problems. In-Reply-To: <61D13DC0-0450-472C-87AB-D2FC312ADDE2@mac.com> References: <32822fe60606302039n2246fbfay53a3fed9bf50d94f@mail.gmail.com> <32822fe60607110817x1465d1c8tec85a28890ad31a7@mail.gmail.com> <32822fe60607140635m55d446c2s97299fd4c807853f@mail.gmail.com> <61D13DC0-0450-472C-87AB-D2FC312ADDE2@mac.com> Message-ID: <32822fe60607242027p5c323750rb0f6bf46f185902d@mail.gmail.com> sorry for not replying earlier I have been busy at work. On 7/14/06, Ronald Oussoren wrote: > > > On Jul 14, 2006, at 3:35 PM, Jorge Vargas wrote: > > > On 7/11/06, Ronald Oussoren wrote: > > > > On Jul 11, 2006, at 5:17 PM, Jorge Vargas wrote: > > > > > noone cares about this? it's a bug so simple to fix.... > > > > There's loads of bugs and patches in the python tracker, and only a > > limited amount of time that people work on python. The patch you > > mentioned also sounds like a fix for a very limited audience, which > > could explain why nobody has seriously looked at it yet. > > > > the patch is more then a year old and is so simple yet so usefull > > it will take less then 10 min for someone that has commit access > > I haven't looked at the actual patch yet, just read its description. > Given its description this patch is not something I would have > reviewed because it seems to be for a very limited audience and is > not something I can test without additional work (e.g. installing > ccache). If you open the patch it's just adding one flag to gcc > > > Are there systems other then gentoo that don't use gcc and g++ but > > some seemingly cross-compiling setup? > > > > the patch is directed to the unixcompiler so why cross-compile? > > Your example mentions a gcc that is named i686-pc-linux-gnu-gcc, > which looks like a cross-compiling setup to me, a conventional gcc > setup just has 'gcc' as the name of the compiler. Which systems > install gcc as i686-pc-linux-gnu-gcc? the system is gentoo the reason for ccache is that gentoo is a source based distro. As for the name it is just how the package system (emerge) calls it, although it will let you compile for other archs, the defaults are set to your system at instalation time. gcc -v Reading specs from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.6/specs Configured with: /var/tmp/portage/gcc-3.3.6/work/gcc-3.3.6/configure --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/3.3.6 --includedir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.6/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.6 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.6/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.6/info --with-gxx-include-dir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.6/include/g++-v3 --host=i686-pc-linux-gnu --build=i686-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --disable-libunwind-exceptions --disable-multilib --disable-libgcj --enable-languages=c,c++,f77 --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu Thread model: posix gcc version 3.3.6 (Gentoo 3.3.6, ssp-3.3.6-1.0, pie-8.7.8) Ronald > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060724/61284493/attachment.htm From mike_mp at zzzcomputing.com Fri Jul 28 02:36:52 2006 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Thu, 27 Jul 2006 20:36:52 -0400 Subject: [Distutils] unusual platform-dependent problem with tarfile created by setuptools Message-ID: <82ADCC51-B23B-47BE-A300-4B721AE41282@zzzcomputing.com> hey all - the MyghtyUtils package, version 0.52 on cheeseshop, has this general file structure: MyghtyUtils-0.52/setup.py MyghtyUtils-0.52/ez_setup.py MyghtyUtils-0.52/lib/myghtyutils/__init__.py MyghtyUtils-0.52/lib/myghtyutils/memcached.py MyghtyUtils-0.52/lib/myghtyutils/ MyghtyUtils-0.52/test/Container.py MyghtyUtils-0.52/test/LRUCache.py MyghtyUtils-0.52/test/SyncDict.py MyghtyUtils-0.52/test/testbase.py however, when you run "python setup.py sdist" on an OS X machine (interestingly, *not* on a windows machine), the resulting tar/gz file contains an extra file: MyghtyUtils-0.52/test/._Container.py no idea why that is...but this creates problems on windows. Then, when you go to install this package on a windows machine (*not* on an OS X or other unixy machine), you get this error: File "c:\python24\lib\site-packages\setuptools-0.6c1-py2.4.egg\setuptools \archive_util.py", line 192, in unpack_tarfile tarobj._extract_member(member,dst) # XXX Ugh File "C:\Python24\lib\tarfile.py", line 1440, in _extract_member self.utime(tarinfo, targetpath) File "C:\Python24\lib\tarfile.py", line 1572, in utime raise ExtractError, "could not change modification time" tarfile.ExtractError: could not change modification time when I go into tarfile.py line 1572 and modify the exception to also print out the underying error, you get this: IOError: [Errno 13] Permission Denied: 'MyghtyUtils-0.52\\test\ \._Container.py' So the extra "._Container.py" file is the source of the windows issue. If you want to reproduce, simply check out the latest source of MyghtyUtils from http://svn.myghty.org/myghtyutils/trunk and run "setup.py sdist". I havent seen this happen with my other packages. I have solved the problem by just running the sdist on the windows machine. but this is pretty strange. any ideas ? From ronaldoussoren at mac.com Fri Jul 28 20:38:56 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 28 Jul 2006 20:38:56 +0200 Subject: [Distutils] unusual platform-dependent problem with tarfile created by setuptools In-Reply-To: <82ADCC51-B23B-47BE-A300-4B721AE41282@zzzcomputing.com> References: <82ADCC51-B23B-47BE-A300-4B721AE41282@zzzcomputing.com> Message-ID: On Jul 28, 2006, at 2:36 AM, Michael Bayer wrote: > MyghtyUtils-0.52/test/testbase.py > > however, when you run "python setup.py sdist" on an OS X machine > (interestingly, *not* on a windows machine), the resulting tar/gz > file contains an extra file: > > MyghtyUtils-0.52/test/._Container.py > > no idea why that is...but this creates problems on windows. test/Container.py contains a resource-fork or some other mac-specific attribute. At least in plain distutils 'sdist' seems to the platform tar and on OSX that creates ._* files for mac-specific attributes/ resource-forks. Ronald From mike_mp at zzzcomputing.com Sat Jul 29 06:31:57 2006 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sat, 29 Jul 2006 00:31:57 -0400 (EDT) Subject: [Distutils] unusual platform-dependent problem with tarfile createdby setuptools In-Reply-To: References: <82ADCC51-B23B-47BE-A300-4B721AE41282@zzzcomputing.com> Message-ID: <10214.216.107.227.217.1154147517.squirrel@www.geekisp.com> mac specific ? i dont think so. however, perhaps something in the file is being incorrectly construed as mac-specific (but still, I dont think so). heres the code: http://www.myghty.org/trac/browser/myghtyutils/trunk/test/Container.py and the full set of imports... from myghtyutils.container import * import myghtyutils.buffer as buffer import random, time, weakref, sys, re import testbase import unittest, sys try: import thread except: raise "this test requires a thread-enabled python" and from then on its just unit tests. Ill see if just having a file called "Container.py" with minimal python in it produces the same results (not at an appropriate workstation at the moment). Ronald Oussoren wrote: > > On Jul 28, 2006, at 2:36 AM, Michael Bayer wrote: >> MyghtyUtils-0.52/test/testbase.py >> >> however, when you run "python setup.py sdist" on an OS X machine >> (interestingly, *not* on a windows machine), the resulting tar/gz >> file contains an extra file: >> >> MyghtyUtils-0.52/test/._Container.py >> >> no idea why that is...but this creates problems on windows. > > test/Container.py contains a resource-fork or some other mac-specific > attribute. At least in plain distutils 'sdist' seems to the platform > tar and on OSX that creates ._* files for mac-specific attributes/ > resource-forks. > > Ronald > > From ronaldoussoren at mac.com Sat Jul 29 08:35:32 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 29 Jul 2006 08:35:32 +0200 Subject: [Distutils] unusual platform-dependent problem with tarfile createdby setuptools In-Reply-To: <10214.216.107.227.217.1154147517.squirrel@www.geekisp.com> References: <82ADCC51-B23B-47BE-A300-4B721AE41282@zzzcomputing.com> <10214.216.107.227.217.1154147517.squirrel@www.geekisp.com> Message-ID: <8D6D1093-370B-4DAA-AD73-C135745F813C@mac.com> On Jul 29, 2006, at 6:31 AM, Michael Bayer wrote: > mac specific ? i dont think so. however, perhaps something in > the file > is being incorrectly construed as mac-specific (but still, I dont > think > so). > > heres the code: That's not relevant, that is the data fork of the file. The file has some non-posix attributes (such as the attribute that gets set when you set the 'Locked' flag in the 'Get Info' screen info the Finder), or a resource fork (maybe it has a custom Icon). I can't tell which without actually seeing the data. Feel free to sent me the output of sdist on the mac and I'll have a look. Ronald From mike_mp at zzzcomputing.com Sun Jul 30 05:41:06 2006 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Sat, 29 Jul 2006 23:41:06 -0400 (EDT) Subject: [Distutils] unusual platform-dependent problem with tarfile createdby setuptools In-Reply-To: <8D6D1093-370B-4DAA-AD73-C135745F813C@mac.com> References: <82ADCC51-B23B-47BE-A300-4B721AE41282@zzzcomputing.com> <10214.216.107.227.217.1154147517.squirrel@www.geekisp.com> <8D6D1093-370B-4DAA-AD73-C135745F813C@mac.com> Message-ID: <10711.216.107.227.217.1154230866.squirrel@www.geekisp.com> sorry, I wasnt familiar with the term "resource fork" or its specifics until now, post-wiki/googling. so yes this is exactly the issue. quickest way to whack it, short of downloading all kinds of resource-fork-management tools (theres also a bunch that come with xcode but i cant get them to do anything useful...such as DeRez which just says the resource fork is "empty and unitialized"...why is that?), seems to be just cat the text of the file so some other file, delete the original, then rename; then youre just left with the data fork and that seems to clean it up. thanks very much. Ronald Oussoren wrote: > > On Jul 29, 2006, at 6:31 AM, Michael Bayer wrote: > >> mac specific ? i dont think so. however, perhaps something in >> the file >> is being incorrectly construed as mac-specific (but still, I dont >> think >> so). >> >> heres the code: > > That's not relevant, that is the data fork of the file. The file has > some non-posix attributes (such as the attribute that gets set when > you set the 'Locked' flag in the 'Get Info' screen info the Finder), > or a resource fork (maybe it has a custom Icon). > > I can't tell which without actually seeing the data. Feel free to > sent me the output of sdist on the mac and I'll have a look. > > Ronald > > From sergei.emantayev at gmail.com Sun Jul 30 10:43:05 2006 From: sergei.emantayev at gmail.com (Sergei Emantayev) Date: Sun, 30 Jul 2006 11:43:05 +0300 Subject: [Distutils] A question about installing Distutils Message-ID: Hello, I'm trying to install the Distutils libarary since it does not included in my python installation. I run Suse linux. I have python 2.4. I've downloaded the distutils version 1.0.2 (the latest one). When I run 'python setup.pyinstall' from the installation directory, I get the following error: Traceback (most recent call last): File "setup.py", line 30, in ? packages = ['distutils', 'distutils.command'], File "/home/semantay/Distutils-1.0.2/distutils/core.py", line 101, in setup _setup_distribution = dist = klass(attrs) File "/home/semantay/Distutils-1.0.2/distutils/dist.py", line 130, in __init__ setattr(self, method_name, getattr(self.metadata, method_name)) AttributeError: DistributionMetadata instance has no attribute 'get___doc__' Please help me. Sergey -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060730/69f9d7fb/attachment.htm From ronaldoussoren at mac.com Sun Jul 30 11:24:30 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 30 Jul 2006 11:24:30 +0200 Subject: [Distutils] A question about installing Distutils In-Reply-To: References: Message-ID: <36C68E51-BEE6-4586-A212-AA5090B98139@mac.com> On Jul 30, 2006, at 10:43 AM, Sergei Emantayev wrote: > Hello, > > I'm trying to install the Distutils libarary since it does not > included in my python installation. I run Suse linux. I have python > 2.4. I've downloaded the distutils version 1.0.2 (the latest one). > When I run 'python setup.py install' from the installation > directory, I get the following error: I can't help you with the error you're getting, but if you want to use distutils on most linux distributions you should install the development package for python, commonly named 'python-dev' or 'python-devel'. Ronald From sergei.emantayev at gmail.com Sun Jul 30 11:39:20 2006 From: sergei.emantayev at gmail.com (Sergei Emantayev) Date: Sun, 30 Jul 2006 12:39:20 +0300 Subject: [Distutils] A question about installing Distutils In-Reply-To: <36C68E51-BEE6-4586-A212-AA5090B98139@mac.com> References: <36C68E51-BEE6-4586-A212-AA5090B98139@mac.com> Message-ID: On 7/30/06, Ronald Oussoren wrote: > > On Jul 30, 2006, at 10:43 AM, Sergei Emantayev wrote: > > > Hello, > > > > I'm trying to install the Distutils libarary since it does not > > included in my python installation. I run Suse linux. I have python > > 2.4. I've downloaded the distutils version 1.0.2 (the latest one). > > When I run 'python setup.py install' from the installation > > directory, I get the following error: > > I can't help you with the error you're getting, but if you want to > use distutils on most linux distributions you should install the > development package for python, commonly named 'python-dev' or > 'python-devel'. > Thank you Ronald for the infromation. I have installed python-devel now and distutils appears inside this package. > Ronald > > Sergei From fdjcomp at yahoo.com Mon Jul 31 10:39:36 2006 From: fdjcomp at yahoo.com (fredaa elder) Date: Mon, 31 Jul 2006 01:39:36 -0700 (PDT) Subject: [Distutils] module_path, recursive scanning for (.pth) files does not work Message-ID: <20060731083936.37939.qmail@web36604.mail.mud.yahoo.com> Hi I have a problem (possible bug?) with the search path setup: version python 2.4.3 located in c:\python I want to have my modules in a completely separate dir: f:\dev\python\lib i made a test module f:\dev\python\lib\math\test_module.py now i have a test file in f:\dev\python\lib\test\test_001.py which needs to load test_module.py manual text: Installing Python Modules /Greg Ward Python Software Foundation Email: distutils-sig at python.org Release 2.4.3 29 March 2006 says: The most convenient way is to add a path configuration file to a directory that's already on Python's path, usually to the .../site-packages/ directory. Path configuration files have an extension of .pth, and each line must contain a single path that will be appended to sys.path. (Because the new paths are appended to sys.path, modules in the added directories will not override standard modules. This means you can't use this mechanism for installing fixed versions of standard modules.) Paths can be absolute or relative, in which case they're relative to the directory containing the .pth file. Any directories added to the search path will be scanned in turn for .pth files So: I put a file "lib.pth" in ../site-packages/ If I put the full path (f:\dev\python\lib\math) to test_module in this file, it works and test_001.py can load the test_module Now for the problem If i put only (f:\dev\python\lib) in lib.pth and I put a file math.pth in this path with a relative dir (math) sys.path will only have (f:\dev\python\lib) added to it and test_001.py cannot load the test_module conclusion: the "Any directories added to the search path will be scanned in turn for .pth files" does not work any suggestions? what did i do wrong? hope you can point me the way and thanks for a great language Frederik AA de Jonge __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From faassen at infrae.com Mon Jul 31 20:15:57 2006 From: faassen at infrae.com (Martijn Faassen) Date: Mon, 31 Jul 2006 20:15:57 +0200 Subject: [Distutils] eggs and Python Unicode variants (UCS2, UCS4) Message-ID: <44CE48DD.3020000@infrae.com> Hi there, In the lxml project (http://codespeak.net/lxml), we've just noticed the following problem with lxml eggs: you can easy_install an egg that won't work for your Python. This is because Python can be compiled with either 2 or 4 bytes unicode as its internal representation. Any egg that contains compiled C code that uses unicode such as lxml will run into trouble: if it's compiled with a 4 bytes unicode Python, it won't work on a 2 bytes unicode Python, and vice versa. This problem is fairly common in Linux. Many distributions such as Ubuntu and Fedora compile their python with 4 bytes unicode internal representation. If you compile a Python interpreter by hand it defaults to 2 bytes unicode, however. Hand-building a Python interpreter is done fairly commonly by Linux sysadmins for various reasons. It would therefore be very nice if it became possible to make eggs for the different unicode compilation options of Python. This configuration dimension is a real world issue for any binary Python module that does anything with unicode text.. In an earlier mail to this list: http://mail.python.org/pipermail/distutils-sig/2005-October/005222.html M.-A. Lemburg and Phillip Eby had the following discussion: [MAL] >>Please make sure that your eggs catch all possible >>Python binary build dimensions: >> >>* Python version >>* Python Unicode variant (UCS2, UCS4) >>* OS name >>* OS version >>* Platform architecture (e.g. 32-bit vs. 64-bit) [PJE] >As far as I know, all of this except the Unicode variant is captured in >distutils' get_platform(). And if it's not, it should be, since it >affects any other kind of bdist mechanism. I'm not sure whether this means this needs to be escalated from setuptools to the Python interpreter level itself. With this mail, I've done the job escalating this lxml problem to what appears to be the right place, though. :) Thanks, Martijn