From smurf at smurf.noris.de Mon Jan 2 08:45:28 2006 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Mon, 2 Jan 2006 08:45:28 +0100 Subject: [Distutils] Setuptools' goal in rpm/deb-based systems In-Reply-To: <5.1.1.6.0.20051229132412.04254c58@mail.telecommunity.com> References: <6b9c17630512290705m22a4a9f1i4bfb8439a250e227@mail.gmail.com> <5.1.1.6.0.20051229132412.04254c58@mail.telecommunity.com> Message-ID: <20060102074528.GG17399@kiste.smurf.noris.de> Hi, Phillip J. Eby: > FYI, unless there's some policy preventing it, you can use the current SVN > snapshot as a basis for a Debian package of setuptools. At the moment (i.e., with no release due soon), Debian Sid should be happy with that. > pre-release tags. If all else fails, you could treat it as > "0.6a8-r41780" instead, which might be a more compatible numbering > approach. > Debian Unstable supports "0.6a9~r41780" as a way to state earlier-than- 0.6a9-ness. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - But you who live on dreams, you are better pleased with the sophistical reasoning and frauds of talkers about great and uncertain matters than those who speak of certain and natural matters, not of such lofty nature. -- Leonardo Da Vinci, "The Codex on the Flight of Birds" -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060102/417736ec/attachment.pgp From pje at telecommunity.com Wed Jan 4 19:15:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 04 Jan 2006 13:15:39 -0500 Subject: [Distutils] setuptools 0.6a9 is released Message-ID: <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> This version contains mostly bug fixes and robustness improvements in a variety of areas, ranging from improved web spidering to better handling of spaces in the Windows directory names for generated .exe files. The biggest new feature included, however, is system packager support including improved bdist_* command handling, and support for installing projects in a "single version, externally managed" format. You can now also build RPMs and Windows installers of setuptools-baseds project without any special steps, even if the project uses egg metadata. The other new features are for developers of setuptools-based projects, and mostly have to do with improved handling of source distribution manifest generation and specifying package data to be included in a project. You can now simply specify "include_package_data=True" and any file under revision control or listed in your source manifest (or MANIFEST.in) will be installed as part of the package. There is also an 'exclude_package_data' option that can be used to trim back the list, and the 'package_data' option can still be used to add files that aren't in revision control or the source manifest. Finally, there have been many documentation updates, including an overhaul of the installation instructions and a new "What your users should know" section of the setuptools manual for project developers. The next release of setuptools will be 0.6a10, for which I plan to add shared library building support (it's needed for an OSAF project, PyICU). There are also some minor features (like dependency_links) that didn't make it into 0.6a9 but which aren't big enough to need waiting till 0.7. Special thanks to Kevin Dangoor, Ian Bicking, and everyone else who reported bugs or helped track them down. Thanks to your participation, 0.6a9 is in much better shape than 0.6a8, and I look forward to a great 0.6a10! From ianb at colorstudy.com Wed Jan 4 19:42:33 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 04 Jan 2006 12:42:33 -0600 Subject: [Distutils] setuptools 0.6a9 is released In-Reply-To: <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> References: <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> Message-ID: <43BC1719.4090605@colorstudy.com> Phillip J. Eby wrote: > The next release of setuptools will be 0.6a10, for which I plan to add > shared library building support (it's needed for an OSAF project, > PyICU). There are also some minor features (like dependency_links) that > didn't make it into 0.6a9 but which aren't big enough to need waiting till 0.7. How about logging of some sort? I'm frequently encountering problems now where I don't understand why there's a conflict or why something is being required. I attribute this to a larger network of packages and dependencies, and handling more versions of software. Mostly the information I'm looking for is a path of requirements -- what caused what else to be required, and in what order. I see that "VersionConflict / XXX add more info" message a lot too, so that's the XXX I'd like to see fixed. I can submit a patch for some of this if you can advise how you'd like to see that work. The logging module? Just a verbose or debug mode of some sort? Though post-mortem why-did-I-get-this-conflict information would be really nice too, which would imply tracking the information all the time instead of just in a debugging mode. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Jan 4 20:00:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 04 Jan 2006 14:00:07 -0500 Subject: [Distutils] setuptools 0.6a9 is released In-Reply-To: <43BC1719.4090605@colorstudy.com> References: <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060104134847.0223bf10@mail.telecommunity.com> At 12:42 PM 1/4/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>The next release of setuptools will be 0.6a10, for which I plan to add >>shared library building support (it's needed for an OSAF project, >>PyICU). There are also some minor features (like dependency_links) that >>didn't make it into 0.6a9 but which aren't big enough to need waiting till 0.7. > >How about logging of some sort? I'm frequently encountering problems now >where I don't understand why there's a conflict or why something is being >required. I attribute this to a larger network of packages and >dependencies, and handling more versions of software. Mostly the >information I'm looking for is a path of requirements -- what caused what >else to be required, and in what order. I see that "VersionConflict / XXX >add more info" message a lot too, so that's the XXX I'd like to see fixed. > >I can submit a patch for some of this if you can advise how you'd like to >see that work. The logging module? Just a verbose or debug mode of some >sort? Though post-mortem why-did-I-get-this-conflict information would be >really nice too, which would imply tracking the information all the time >instead of just in a debugging mode. The reason I have the "add more info" comment is that resolve() could track what distributions had what requirements, so the error could potentially be something like: VersionConflict: foo 1.2 conflicts with foo>=1.3 needed by bar 1.7, spam 2.1 while resolving dependencies for SomeProject>0.8 If that's all you're looking for, it wouldn't be hard to make resolve() do it. Of course, that doesn't tell you the full story if there are more layers of dependencies, but I think maybe a detailed dependency analysis tool should be part of the "nest" command suite in the setuptools 0.7 series. Something like "nest analyze 'SomeProject>0.8'" to dump out all the dependencies and how they are currently being met and/or conflicting. From ianb at colorstudy.com Wed Jan 4 20:23:37 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 04 Jan 2006 13:23:37 -0600 Subject: [Distutils] setuptools 0.6a9 is released In-Reply-To: <5.1.1.6.0.20060104134847.0223bf10@mail.telecommunity.com> References: <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> <5.1.1.6.0.20060104134847.0223bf10@mail.telecommunity.com> Message-ID: <43BC20B9.70001@colorstudy.com> Phillip J. Eby wrote: > If that's all you're looking for, it wouldn't be hard to make resolve() > do it. Of course, that doesn't tell you the full story if there are > more layers of dependencies, but I think maybe a detailed dependency > analysis tool should be part of the "nest" command suite in the > setuptools 0.7 series. Something like "nest analyze 'SomeProject>0.8'" > to dump out all the dependencies and how they are currently being met > and/or conflicting. Since I'm requiring packages from places other than requires.txt (e.g., config files) the nest command seems like it would be limited in what it could provide. Not useless, but limited. Runtime logging would be more complete in those situations. Or maybe just some way to indicate what requirements a config file implies, or otherwise wrap that analysis. If I want to analyze a paste.deploy config file, it really comes down to parsing out each section, and creating a list of requirements for each section, and then displaying the analysis results of that list. Then I'm guessing there would be an entirely different command that would analyze paste.deploy config files. Any particular plans for nest? We'd talked briefly about paster serving as a basis (with some portions removed and put in paste.deploy, where they probably already belong). Mostly it's just command-line UI, but the one more architectural issue is how plugins are picked up. I'm pretty happy with the .egg-info config file (paster_plugins.txt) that lists requirements (frameworks, typically) of a package, and entry points are brought in based on that. Though, now that I think about it, requires.txt might be perfectly usable for that (recursively bringing in all requirements, or just the immediate requirements? -- I'm thinking just immediate). Anyway, this applies to a package in development; I haven't really thought through how it applies to an installed package. Commands could still be installed globally, but for many commands that's not appropriate. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ronaldoussoren at mac.com Wed Jan 4 21:51:44 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 4 Jan 2006 21:51:44 +0100 Subject: [Distutils] [setuptools] setuptools and fat binaries Message-ID: <4034E57A-36E9-48ED-B6F2-C9A4AFF62BA9@mac.com> Hi, Apple supports fat binaries on Mac OS X (they call them universal binaries), that is binaries that contain executable code for multiple architectures. In released version of the os this can be used to build binaries and libraries that support both PPC and PPC64 architectures, which isn't used very much, and in the future this will be the mechanism to support both the current PPC based systems and the to-be-introduced intel systems with a single binary. I'm currently playing around with Python on an Intel developer system (my goal is a patch that will make it easy to build a fat/universal build of Python on Mac OS X), and I wondering how setuptools (and specifically eggs) can support universal binaries. The reason I ask is because of eggs that contain extensions, whose name contains the architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just using two eggs, one for each architecture, would work but seems wasteful and would make it harder to create self fat application bundles. Ronald From bob at redivi.com Wed Jan 4 22:26:00 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 4 Jan 2006 13:26:00 -0800 Subject: [Distutils] [setuptools] setuptools and fat binaries In-Reply-To: <4034E57A-36E9-48ED-B6F2-C9A4AFF62BA9@mac.com> References: <4034E57A-36E9-48ED-B6F2-C9A4AFF62BA9@mac.com> Message-ID: <42EECB35-6A9E-4E14-9468-86AB6E19E29A@redivi.com> On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote: > Apple supports fat binaries on Mac OS X (they call them universal > binaries), that is binaries that contain executable code for multiple > architectures. In released version of the os this can be used to > build binaries and libraries that support both PPC and PPC64 > architectures, which isn't used very much, and in the future this > will be the mechanism to support both the current PPC based systems > and the to-be-introduced intel systems with a single binary. > > I'm currently playing around with Python on an Intel developer system > (my goal is a patch that will make it easy to build a fat/universal > build of Python on Mac OS X), and I wondering how setuptools (and > specifically eggs) can support universal binaries. The reason I ask > is because of eggs that contain extensions, whose name contains the > architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just > using two eggs, one for each architecture, would work but seems > wasteful and would make it harder to create self fat application > bundles. Using two or three eggs doesn't really make it any harder to create an application bundle, py2app needs to change to support eggs anyway and having a separate arch egg would work fine. It might be better that way because it's easy to see which packages support which architectures and it's easy for someone who has the relevant platform to built, test, and upload. py2app could even merge arch eggs together to make them fat at build time. The only thing that's off is that easy_install would need to fetch eggs for all architectures... maybe a pydistutils.cfg that says you want to fetch "ppc ppc64 x86". -bob From ronaldoussoren at mac.com Wed Jan 4 22:38:09 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 4 Jan 2006 22:38:09 +0100 Subject: [Distutils] [setuptools] setuptools and fat binaries In-Reply-To: <42EECB35-6A9E-4E14-9468-86AB6E19E29A@redivi.com> References: <4034E57A-36E9-48ED-B6F2-C9A4AFF62BA9@mac.com> <42EECB35-6A9E-4E14-9468-86AB6E19E29A@redivi.com> Message-ID: On 4-jan-2006, at 22:26, Bob Ippolito wrote: > On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote: > >> Apple supports fat binaries on Mac OS X (they call them universal >> binaries), that is binaries that contain executable code for multiple >> architectures. In released version of the os this can be used to >> build binaries and libraries that support both PPC and PPC64 >> architectures, which isn't used very much, and in the future this >> will be the mechanism to support both the current PPC based systems >> and the to-be-introduced intel systems with a single binary. >> >> I'm currently playing around with Python on an Intel developer system >> (my goal is a patch that will make it easy to build a fat/universal >> build of Python on Mac OS X), and I wondering how setuptools (and >> specifically eggs) can support universal binaries. The reason I ask >> is because of eggs that contain extensions, whose name contains the >> architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just >> using two eggs, one for each architecture, would work but seems >> wasteful and would make it harder to create self fat application >> bundles. > > Using two or three eggs doesn't really make it any harder to create > an application bundle, py2app needs to change to support eggs > anyway and having a separate arch egg would work fine. It might be > better that way because it's easy to see which packages support > which architectures and it's easy for someone who has the relevant > platform to built, test, and upload. Someone, somewhere has to do the work, either setuptools or py2app. I'd like to see fat eggs to avoid duplication of all non-executable stuff (python files and data files). I'd guess that in most cases just using '-arch i386 -arch ppc' in the extra_compile_args and extra_link_args would enable you to build a fat version of an extension. Building a fat egg that works on OSX 10.3 and forward might be harder, but even there the simple solution might work most of the time (I haven't put much thought into that yet). > > py2app could even merge arch eggs together to make them fat at > build time. If it would do that setuptools would still require some support for fat eggs, or do you mean that py2app would merge the two variants and then have a symlink from one architecture variant to the other? > The only thing that's off is that easy_install would need to fetch > eggs for all architectures... maybe a pydistutils.cfg that says you > want to fetch "ppc ppc64 x86". > > -bob > From bob at redivi.com Wed Jan 4 22:48:33 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 4 Jan 2006 13:48:33 -0800 Subject: [Distutils] [setuptools] setuptools and fat binaries In-Reply-To: References: <4034E57A-36E9-48ED-B6F2-C9A4AFF62BA9@mac.com> <42EECB35-6A9E-4E14-9468-86AB6E19E29A@redivi.com> Message-ID: On Jan 4, 2006, at 1:38 PM, Ronald Oussoren wrote: > > On 4-jan-2006, at 22:26, Bob Ippolito wrote: > >> On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote: >> >>> Apple supports fat binaries on Mac OS X (they call them universal >>> binaries), that is binaries that contain executable code for >>> multiple >>> architectures. In released version of the os this can be used to >>> build binaries and libraries that support both PPC and PPC64 >>> architectures, which isn't used very much, and in the future this >>> will be the mechanism to support both the current PPC based systems >>> and the to-be-introduced intel systems with a single binary. >>> >>> I'm currently playing around with Python on an Intel developer >>> system >>> (my goal is a patch that will make it easy to build a fat/universal >>> build of Python on Mac OS X), and I wondering how setuptools (and >>> specifically eggs) can support universal binaries. The reason I ask >>> is because of eggs that contain extensions, whose name contains the >>> architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just >>> using two eggs, one for each architecture, would work but seems >>> wasteful and would make it harder to create self fat application >>> bundles. >> >> Using two or three eggs doesn't really make it any harder to >> create an application bundle, py2app needs to change to support >> eggs anyway and having a separate arch egg would work fine. It >> might be better that way because it's easy to see which packages >> support which architectures and it's easy for someone who has the >> relevant platform to built, test, and upload. > > Someone, somewhere has to do the work, either setuptools or py2app. > I'd like to see fat eggs to avoid duplication of all non-executable > stuff (python files and data files). Work has to be done in both places, either way. > I'd guess that in most cases just using '-arch i386 -arch ppc' in > the extra_compile_args and extra_link_args would enable you to > build a fat version of an extension. Building a fat egg that works > on OSX 10.3 and forward might be harder, but even there the simple > solution might work most of the time (I haven't put much thought > into that yet). Aren't there still pyconfig.h issues with doing a fat build like that? What about getting ppc64 support in a fat build? >> >> py2app could even merge arch eggs together to make them fat at >> build time. > > If it would do that setuptools would still require some support for > fat eggs, or do you mean that py2app would merge the two variants > and then have a symlink from one architecture variant to the other? There are lots of ways it could work. It doesn't really need any support of fat eggs, it could just remove the platform spec. -bob From ronaldoussoren at mac.com Wed Jan 4 23:04:04 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 4 Jan 2006 23:04:04 +0100 Subject: [Distutils] [setuptools] setuptools and fat binaries In-Reply-To: References: <4034E57A-36E9-48ED-B6F2-C9A4AFF62BA9@mac.com> <42EECB35-6A9E-4E14-9468-86AB6E19E29A@redivi.com> Message-ID: <666349E7-6C97-44DE-8631-6EB721B67C8C@mac.com> On 4-jan-2006, at 22:48, Bob Ippolito wrote: > > On Jan 4, 2006, at 1:38 PM, Ronald Oussoren wrote: > >> >> On 4-jan-2006, at 22:26, Bob Ippolito wrote: >> >>> On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote: >>> >>>> Apple supports fat binaries on Mac OS X (they call them universal >>>> binaries), that is binaries that contain executable code for >>>> multiple >>>> architectures. In released version of the os this can be used to >>>> build binaries and libraries that support both PPC and PPC64 >>>> architectures, which isn't used very much, and in the future this >>>> will be the mechanism to support both the current PPC based systems >>>> and the to-be-introduced intel systems with a single binary. >>>> >>>> I'm currently playing around with Python on an Intel developer >>>> system >>>> (my goal is a patch that will make it easy to build a fat/universal >>>> build of Python on Mac OS X), and I wondering how setuptools (and >>>> specifically eggs) can support universal binaries. The reason I ask >>>> is because of eggs that contain extensions, whose name contains the >>>> architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just >>>> using two eggs, one for each architecture, would work but seems >>>> wasteful and would make it harder to create self fat application >>>> bundles. >>> >>> Using two or three eggs doesn't really make it any harder to >>> create an application bundle, py2app needs to change to support >>> eggs anyway and having a separate arch egg would work fine. It >>> might be better that way because it's easy to see which packages >>> support which architectures and it's easy for someone who has the >>> relevant platform to built, test, and upload. >> >> Someone, somewhere has to do the work, either setuptools or >> py2app. I'd like to see fat eggs to avoid duplication of all non- >> executable stuff (python files and data files). > > Work has to be done in both places, either way. > >> I'd guess that in most cases just using '-arch i386 -arch ppc' in >> the extra_compile_args and extra_link_args would enable you to >> build a fat version of an extension. Building a fat egg that works >> on OSX 10.3 and forward might be harder, but even there the simple >> solution might work most of the time (I haven't put much thought >> into that yet). > > Aren't there still pyconfig.h issues with doing a fat build like > that? What about getting ppc64 support in a fat build? My current hack is to include macosx-config.h at the end of pyconfig.h and to redefine all processor specific defines in that file. That seems to allow me to build a fat framework using 'OPT=- arch i386 -arch ppc LDFLAGS=-arch i386 -arch ppc'. Well, actually that and a patch to Makefile.pre.in to call libtool in the right way. But, I haven't tested the result on a PPC box yet, I'm hunting down i386 specific bugs first (probably byte-order related). Adding ppc64 support would require 3 seperate builds + a merge because not all extensions build on the ppc64 flavor, the entire Carbon package cannot be build for ppc64. I don't have a G5 box, and probably would work on this even if I had. Making it possible to build a 10.3 capable python binary on 10.4 is more useful. > >>> >>> py2app could even merge arch eggs together to make them fat at >>> build time. >> >> If it would do that setuptools would still require some support >> for fat eggs, or do you mean that py2app would merge the two >> variants and then have a symlink from one architecture variant to >> the other? > > There are lots of ways it could work. It doesn't really need any > support of fat eggs, it could just remove the platform spec. > > -bob > P.S. the ruckus(sp?) about a compiler warning getpath.c over on python-dev is quite funny given the amount of warnings I'm getting in other parts of the build, although most of those are in Carbon package. From pje at telecommunity.com Thu Jan 5 02:37:35 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 04 Jan 2006 20:37:35 -0500 Subject: [Distutils] setuptools 0.6a9 is released In-Reply-To: <43BC20B9.70001@colorstudy.com> References: <5.1.1.6.0.20060104134847.0223bf10@mail.telecommunity.com> <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> <5.1.1.6.0.20060104125829.023646d8@mail.telecommunity.com> <5.1.1.6.0.20060104134847.0223bf10@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060104203216.0223e700@mail.telecommunity.com> At 01:23 PM 1/4/2006 -0600, Ian Bicking wrote: >Phillip J. Eby wrote: >>If that's all you're looking for, it wouldn't be hard to make resolve() >>do it. Of course, that doesn't tell you the full story if there are more >>layers of dependencies, but I think maybe a detailed dependency analysis >>tool should be part of the "nest" command suite in the setuptools 0.7 >>series. Something like "nest analyze 'SomeProject>0.8'" to dump out all >>the dependencies and how they are currently being met and/or conflicting. > >Since I'm requiring packages from places other than requires.txt (e.g., >config files) the nest command seems like it would be limited in what it >could provide. Not useless, but limited. Runtime logging would be more >complete in those situations. Or maybe just some way to indicate what >requirements a config file implies, or otherwise wrap that analysis. Note that the resolve() method of WorkingSet does this - it returns a list of distributions, or raises either VersionConflict or DistributionNotFound. Unlike require(), it does *not* alter the contents of the WorkingSet you call it on, and it also allows you to specify an 'installer' callback that will be invoked for distributions that aren't found. > If I want to analyze a paste.deploy config file, it really comes down > to parsing out each section, and creating a list of requirements for each > section, and then displaying the analysis results of that list. Then I'm > guessing there would be an entirely different command that would analyze > paste.deploy config files. > >Any particular plans for nest? We'd talked briefly about paster serving >as a basis (with some portions removed and put in paste.deploy, where they >probably already belong). Mostly it's just command-line UI, but the one >more architectural issue is how plugins are picked up. I'm pretty happy >with the .egg-info config file (paster_plugins.txt) that lists >requirements (frameworks, typically) of a package, and entry points are >brought in based on that. Though, now that I think about it, requires.txt >might be perfectly usable for that (recursively bringing in all >requirements, or just the immediate requirements? -- I'm thinking just >immediate). Anyway, this applies to a package in development; I haven't >really thought through how it applies to an installed package. Commands >could still be installed globally, but for many commands that's not >appropriate. I'm not sure I followed everything you said, but note that entry point definitions can list "extras" that will be require()'d when the entry point is loaded, and it is recursive. So, I anticipate that some "nest" commands will list extras that would be installed on demand, similar to what buildutils does. So, if for example somebody wrote a web UI for doing package management based on TurboGears or Paste, running "nest webui" could first download and install the needed packages before firing up the UI. :) From david at handysoftware.com Thu Jan 5 04:54:05 2006 From: david at handysoftware.com (David Handy) Date: Wed, 4 Jan 2006 22:54:05 -0500 Subject: [Distutils] [setuptools] When .py files are "data", strange things happen Message-ID: <20060105035405.GA5422@arno2> I want setuptools to install a bunch of sample scripts in a package-relative directory. Each end user will later run a script that copies these to a directory relative to his or her home directory. However, strange things happen to data files that have a .py extension. 1) Saying "include_package_data = True" was not enough to get them included in the egg. The .txt files were included, but the .py files were magically excluded. 2) I overcame the above problem with the following setup parameter: package_data = { # Include all .py files in the data directories, they # are excluded by default. 'cpif.data': ['lander/*.py', 'samples/*.py'], } But now I get a bogus .pyc file generated for each of these sample scripts. These bogus .pyc files get included in the egg. 3) I tried to overcome problem #2 above with the following setup parameter: exclude_package_data = { # The example programs should not be shipped with .pyc files # XXX This is not working, the .pyc files are being created and # included anyway. 'cpif.data': ['lander/*.pyc', 'samples/*.pyc'], } But this didn't work. Putting the following lines in my MANIFEST.in file also didn't work: exclude cpif/data/lander/*.pyc exclude cpif/data/samples/*.pyc I suppose I could turn off .pyc file generation, but that's not what I want. I just want to treat certain .py files as data files. The "data" .py files are in a subdirectory of a python package, but the directory in which they directly reside does not contain an __init__.py file, so they are not directly importable. So I don't think .pyc files should be generated for them in any case. Thanks in advance for ideas/sympathy/fixes, etc. -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/ From jeff.pitman at gmail.com Thu Jan 5 05:51:35 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Thu, 5 Jan 2006 12:51:35 +0800 Subject: [Distutils] [setuptools] When .py files are "data", strange things happen In-Reply-To: <20060105035405.GA5422@arno2> References: <20060105035405.GA5422@arno2> Message-ID: <6b9c17630601042051y44503e94r23d313e3599bae1d@mail.gmail.com> On 1/5/06, David Handy wrote: > I just want to treat certain .py files as data files. The "data" .py files > are in a subdirectory of a python package, but the directory in which they > directly reside does not contain an __init__.py file, so they are not > directly importable. So I don't think .pyc files should be generated for > them in any case. Do they *have* to end in .py? I've seen apps that end in .cfg or .dat and happly execfile them into their program space. This is not to say that setuptools shouldn't be improved. I'm just curious as to the choice of the extension. (syntax highlighting concerns can be alleviated for vim and emacs by using a comment). -- -jeff From pje at telecommunity.com Thu Jan 5 18:07:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 05 Jan 2006 12:07:39 -0500 Subject: [Distutils] [setuptools] When .py files are "data", strange things happen In-Reply-To: <20060105035405.GA5422@arno2> Message-ID: <5.1.1.6.0.20060105120109.022603d8@mail.telecommunity.com> At 10:54 PM 1/4/2006 -0500, David Handy wrote: >I want setuptools to install a bunch of sample scripts in a package-relative >directory. Each end user will later run a script that copies these to a >directory relative to his or her home directory. > >However, strange things happen to data files that have a .py extension. > >1) Saying "include_package_data = True" was not enough to get them included > in the egg. The .txt files were included, but the .py files were > magically excluded. > >2) I overcame the above problem with the following setup parameter: > > package_data = { > # Include all .py files in the data directories, they > # are excluded by default. > 'cpif.data': ['lander/*.py', 'samples/*.py'], > } > > But now I get a bogus .pyc file generated for each of these sample > scripts. These bogus .pyc files get included in the egg. If I understand your use case correctly, I don't see how that hurts anything. The simplest way to solve your problem, however, is to put the samples in a subdirectory of your project's .egg-info directory, and then use the metadata APIs to access them. .py files under .egg-info do not get compiled. Meanwhile, it's not really practical to have .py files inside a package be included, but not compiled. So your three choices are: * Give the files a different extension (e.g. .py.orig or .py.in or .cfg or whatever), which will then trivially work with include_package_data=True * Include them explicitly as .py files, as shown above (and then ignore the irrelevant .pyc files) * Put them in a subdirectory of the project's .egg-info, and then use the pkg_resources metadata APIs at runtime to get access. From pje at telecommunity.com Fri Jan 6 00:19:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 05 Jan 2006 18:19:20 -0500 Subject: [Distutils] [setuptools] When .py files are "data", strange things happen In-Reply-To: <1D0F062A-90C3-48AE-94CF-7ED97E44F3F5@groovie.org> References: <5.1.1.6.0.20060105120109.022603d8@mail.telecommunity.com> <5.1.1.6.0.20060105120109.022603d8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060105181535.02237430@mail.telecommunity.com> At 01:48 PM 1/5/2006 -0800, Ben Bangert wrote: >On Jan 5, 2006, at 9:07 AM, Phillip J. Eby wrote: > >>If I understand your use case correctly, I don't see how that hurts >>anything. > >It hurts a ton if you want to use Paster templates, because you'll >have files ending in .py that aren't meant to be processed by paster, >and _tmpl, all of which are inside non-package data directories, thus >won't get included any other way. You lost me. How does having the .py included and compiled to .pyc hurt anything? .pyc files include a date/time stamp of the .py they were built from, so if you accidentally copy them somewhere and change the corresponding .py file, the .pyc will be ignored and overwritten in the import case, and ignored altogether in an execfile or script-running case. In no scenario that I understand, does the presence of a .pyc file hurt anything. Could you please explain what problem you think it will cause? From david at handysoftware.com Fri Jan 6 01:24:51 2006 From: david at handysoftware.com (David Handy) Date: Thu, 5 Jan 2006 19:24:51 -0500 Subject: [Distutils] [setuptools] When .py files are "data", strange things happen In-Reply-To: <5.1.1.6.0.20060105120109.022603d8@mail.telecommunity.com> References: <20060105035405.GA5422@arno2> <5.1.1.6.0.20060105120109.022603d8@mail.telecommunity.com> Message-ID: <20060106002451.GA11017@arno2> On Thu, Jan 05, 2006 at 12:07:39PM -0500, Phillip J. Eby wrote: > At 10:54 PM 1/4/2006 -0500, David Handy wrote: > >I want setuptools to install a bunch of sample scripts in a > >package-relative > >directory. Each end user will later run a script that copies these to a > >directory relative to his or her home directory. > > > >However, strange things happen to data files that have a .py extension. > > > >1) Saying "include_package_data = True" was not enough to get them included > > in the egg. The .txt files were included, but the .py files were > > magically excluded. > > > >2) I overcame the above problem with the following setup parameter: > > > > package_data = { > > # Include all .py files in the data directories, they > > # are excluded by default. > > 'cpif.data': ['lander/*.py', 'samples/*.py'], > > } > > > > But now I get a bogus .pyc file generated for each of these sample > > scripts. These bogus .pyc files get included in the egg. > > If I understand your use case correctly, I don't see how that hurts > anything. My project is not the typical software component - it is a kit for beginning programmers. They are following the directions in my book. The .py "data" files I refer to are a whole bunch of simple sample programs that users are supposed to refer to as they go through my book. Really, they're documentation more than code. Most of my users won't navigate to the package directory itself, but instead will work with copies of those files in their "MyPrograms" directory. But those who do poke around (like I always do), and know a little bit of Python, would wonder, "why are these single file programs, that are never imported, compiled to .pyc files anyway? Am I missing something?" I didn't want the answer to be "they're compiled because the author didn't try to learn how to use setuptools properly." :) But it's not a huge issue. > > The simplest way to solve your problem, however, is to put the samples in a > subdirectory of your project's .egg-info directory, and then use the > metadata APIs to access them. .py files under .egg-info do not get > compiled. > > Meanwhile, it's not really practical to have .py files inside a package be > included, but not compiled. So your three choices are: > > * Give the files a different extension (e.g. .py.orig or .py.in or .cfg or > whatever), which will then trivially work with include_package_data=True I prefer to keep the names intact to make the packaging itself as transparent as possible. > * Include them explicitly as .py files, as shown above (and then ignore the > irrelevant .pyc files) That's the path of least resistance right now. The welcome script that copies the files to the user's MyPrograms directory will ignore .pyc files. > * Put them in a subdirectory of the project's .egg-info, and then use the > pkg_resources metadata APIs at runtime to get access. > I'll probably do this eventually. Right now I am not installing setuptools with my package, just using it to create the installer. Thanks for the advice! -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/ From pje at telecommunity.com Fri Jan 6 05:21:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 05 Jan 2006 23:21:07 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al Message-ID: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes. This seems to be a bit of a problem, in that it appears to mean there's no sane way to include shared libraries in an egg on any platform other than Windows. It's common practice for Python extensions built for Windows to include shared libraries to avoid having to install them in the Windows system directories, but for Unix-like OSes this isn't really the normal practice anyway. But, in the case where a developer wants to include a shared library as part of the distribution, this remains somewhat problematic. One can of course install the library to a platform-appropriate location ($prefix/lib on most platforms), but to do it from an egg is harder. You can't simply unzip the egg any more, you would have to also move the libraries to a standard installation location, breaking any hope of multi-version support. (And this would have to be able to happen on the fly, when shared libraries and extensions are extracted to the cache directory, if running zipped.) I don't suppose you can modify LD_LIBRARY_PATH on the fly? Any other options? Should I just give up on supporting shared libraries on anything but Windows, perhaps silently converting the shared libraries to static ones? From jeff.pitman at gmail.com Fri Jan 6 05:29:21 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Fri, 6 Jan 2006 12:29:21 +0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> Message-ID: <6b9c17630601052029v731d0135q57b232f039b8aa20@mail.gmail.com> On 1/6/06, Phillip J. Eby wrote: > Hi folks. I just implemented preliminary shared library build support for > setuptools, but I ran into a bit of a snag. I was operating under the > assumption that you could simply put shared libraries in the directory > alongside the extensions that use them, but in practice it turns out that > this only works on Windows. On Linux I found that you could only put > shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar > issue will be found under OS X and other Unixes. Are these just extensions that are imported? Because if they are, then you don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on Linux as long as PYTHONPATH is correct. -- -jeff From robert.kern at gmail.com Fri Jan 6 05:55:28 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 05 Jan 2006 22:55:28 -0600 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <6b9c17630601052029v731d0135q57b232f039b8aa20@mail.gmail.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <6b9c17630601052029v731d0135q57b232f039b8aa20@mail.gmail.com> Message-ID: Jeff Pitman wrote: > Are these just extensions that are imported? Because if they are, then > you don't need to mess with LD_LIBRARY_PATH and you can put them > anywhere on Linux as long as PYTHONPATH is correct. No, they are things like libpng.so that the extension modules use. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pje at telecommunity.com Fri Jan 6 05:59:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 05 Jan 2006 23:59:23 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <6b9c17630601052029v731d0135q57b232f039b8aa20@mail.gmail.co m> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> At 12:29 PM 1/6/2006 +0800, Jeff Pitman wrote: >Are these just extensions that are imported? Because if they are, then you >don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on >Linux as long as PYTHONPATH is correct. -- -jeff Nope. They're shared libraries being built by the distribution itself, as in the case of PyICU, and certain Windows C extensions that need to include the target library (e.g. sqlite) because it's not a "system" library there. It could perhaps be questioned whether it's a good idea to include any other kind of shared library in an extension project, and in fact the common practice to link between Python extensions is to use a Python CObject to wrap an API function pointer table. This neatly bypasses all of the issues... *if* you control the library in question. So far I've done a few experiments with trying to hack LD_LIBRARY_PATH at runtime, use dl.open() with RTLD_GLOBAL, etc. to force finding the right library, but so far it seems like there's no way to get the extensions to consider the opened library to be "the same" as the one they're looking for. (Although it may just be that I'm missing some trick on the build side of the equation.) From robert.kern at gmail.com Fri Jan 6 06:00:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 05 Jan 2006 23:00:09 -0600 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > Hi folks. I just implemented preliminary shared library build support for > setuptools, but I ran into a bit of a snag. I was operating under the > assumption that you could simply put shared libraries in the directory > alongside the extensions that use them, but in practice it turns out that > this only works on Windows. On Linux I found that you could only put > shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar > issue will be found under OS X and other Unixes. On OS X, the headers of the dylib can be rewritten to point to a specific path. IIRC, this can be a relative path. Bob Ippolito's py2app does this kind of rewriting for making relocatable .app bundles. I'm sure he'll chime in soon, so I'm not entirely sure why I bothered replying myself. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From smurf at smurf.noris.de Fri Jan 6 06:20:14 2006 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Fri, 6 Jan 2006 06:20:14 +0100 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> Message-ID: <20060106052014.GZ17399@kiste.smurf.noris.de> Hi, Phillip J. Eby: > It could perhaps be questioned whether it's a good idea to include any > other kind of shared library in an extension project It is not. You will end up with multiple copies of extension libraries, all subtly different, scattered in various .egg directories. You guess what happens if two of these try to use the "same" library (hint: it's not pretty and even less debuggable). You guess what happens, or rather what will not happen, if a security problem is discovered in one of these libraries. Besides, Unix systems these days tend to have the required libraries installed anyway. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - "Pok pok pok, P'kok!" -- Superchicken -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060106/ccd6facc/attachment-0001.pgp From bob at redivi.com Fri Jan 6 06:47:15 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 5 Jan 2006 21:47:15 -0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> Message-ID: <33B36AD8-C99C-45EE-9319-2A42B6718EF6@redivi.com> On Jan 5, 2006, at 9:00 PM, Robert Kern wrote: > Phillip J. Eby wrote: >> Hi folks. I just implemented preliminary shared library build >> support for >> setuptools, but I ran into a bit of a snag. I was operating under >> the >> assumption that you could simply put shared libraries in the >> directory >> alongside the extensions that use them, but in practice it turns >> out that >> this only works on Windows. On Linux I found that you could only put >> shared libraries in a directory on LD_LIBRARY_PATH, and I suspect >> a similar >> issue will be found under OS X and other Unixes. > > On OS X, the headers of the dylib can be rewritten to point to a > specific path. > IIRC, this can be a relative path. Bob Ippolito's py2app does this > kind of > rewriting for making relocatable .app bundles. I'm sure he'll chime > in soon, so > I'm not entirely sure why I bothered replying myself. It's actually the extension that needs the rewriting.. but it's good to rewrite the dylib also. It can be a relative path, but only relative to the executable. You can be relative to the bundle on Mac OS X 10.4+, but it really ought to just make the path static upon installation and remain backwards compatible. In order to make that bulletproof, the linker flag - headerpad_max_install_names should be used. -bob From jeff.pitman at gmail.com Fri Jan 6 07:11:56 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Fri, 6 Jan 2006 14:11:56 +0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> Message-ID: <6b9c17630601052211w68a92637u19f421bbf9140cf@mail.gmail.com> On 1/6/06, Phillip J. Eby wrote: > At 12:29 PM 1/6/2006 +0800, Jeff Pitman wrote: > >Are these just extensions that are imported? Because if they are, then you > >don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on > >Linux as long as PYTHONPATH is correct. -- -jeff > > Nope. They're shared libraries being built by the distribution itself, as > in the case of PyICU, and certain Windows C extensions that need to include > the target library (e.g. sqlite) because it's not a "system" library there. Unless you do something really magical, you're going to find a lot of resistance to this in most Linux distro camps. Gentoo will want to recompile via emerge. Fedora/CentOS will want to grab from yum. Debian will want to grab libs via apt. This is why I made the comment that the focus on Linux should really be all about $HOME on servers where the user does not admin the box. Because, in a sense, you're looking to overwrite or replace existing infrastructure and since eggs/setuptools has no visual into rpm/deb, things will get messy. And, admins, for the most part, don't like messes. -- -jeff From pje at telecommunity.com Fri Jan 6 17:19:51 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Jan 2006 11:19:51 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <6b9c17630601052211w68a92637u19f421bbf9140cf@mail.gmail.com > References: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060106111637.0203ca00@mail.telecommunity.com> At 02:11 PM 1/6/2006 +0800, Jeff Pitman wrote: >Unless you do something really magical, you're going to find a lot of >resistance to this in most Linux distro camps. Gentoo will want to >recompile via emerge. Fedora/CentOS will want to grab from yum. Debian >will want to grab libs via apt. The issue for the PyICU/PyLucene etc. use cases is that there are a whole bunch of SWIG wrappers for different extensions, but with a large common library for type mapping etc. The authors want a shared library for *their own code* that is used by multiple extensions. This is not separately distributed code. This is not the same thing as the Windows use case for building a shared library, although the ability to do it on Windows satisfies this use case too. From bob at redivi.com Fri Jan 6 17:33:38 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 6 Jan 2006 08:33:38 -0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060106111637.0203ca00@mail.telecommunity.com> References: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060106111637.0203ca00@mail.telecommunity.com> Message-ID: On Jan 6, 2006, at 8:19 AM, Phillip J. Eby wrote: > At 02:11 PM 1/6/2006 +0800, Jeff Pitman wrote: >> Unless you do something really magical, you're going to find a lot of >> resistance to this in most Linux distro camps. Gentoo will want to >> recompile via emerge. Fedora/CentOS will want to grab from yum. >> Debian >> will want to grab libs via apt. > > The issue for the PyICU/PyLucene etc. use cases is that there are a > whole > bunch of SWIG wrappers for different extensions, but with a large > common > library for type mapping etc. The authors want a shared library > for *their > own code* that is used by multiple extensions. This is not separately > distributed code. It'd be better if these projects just fixed themselves to do things the "right" way, using a Python extension and a function table. -bob From pje at telecommunity.com Fri Jan 6 17:35:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Jan 2006 11:35:41 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <20060106052014.GZ17399@kiste.smurf.noris.de> References: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> At 06:20 AM 1/6/2006 +0100, Matthias Urlichs wrote: >Phillip J. Eby: > > It could perhaps be questioned whether it's a good idea to include any > > other kind of shared library in an extension project > >It is not. > >You will end up with multiple copies of extension libraries, all subtly >different, scattered in various .egg directories. You guess what happens >if two of these try to use the "same" library (hint: it's not pretty and >even less debuggable). You guess what happens, or rather what will not >happen, if a security problem is discovered in one of these libraries. Um, the "other kind" I'm referring to here are the ones that are exclusively part of the extension in question, e.g. libPyICU. This library is not going to be found anywhere else except in other versions of PyICU, so that issue doesn't really apply here, since only one egg will have extensions loading the library into a given process. >Besides, Unix systems these days tend to have the required libraries >installed anyway. Not libPyICU, since it's a part of PyICU. Similar issues may apply to other multi-extension projects that need to share a large chunk of code. From smurf at smurf.noris.de Fri Jan 6 17:43:31 2006 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Fri, 6 Jan 2006 17:43:31 +0100 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> References: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> Message-ID: <20060106164331.GE17399@kiste.smurf.noris.de> Hi, Phillip J. Eby: > Um, the "other kind" I'm referring to here are the ones that are > exclusively part of the extension in question, e.g. libPyICU. This library > is not going to be found anywhere else except in other versions of PyICU, > so that issue doesn't really apply here, since only one egg will have > extensions loading the library into a given process. So why don't you simply link the Python extension and the library to one single .so file? -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - BOFH excuse #263: It's stuck in the Web. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060106/aac1f047/attachment.pgp From pje at telecommunity.com Fri Jan 6 17:58:27 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Jan 2006 11:58:27 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <20060106164331.GE17399@kiste.smurf.noris.de> References: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060106115725.02014b18@mail.telecommunity.com> At 05:43 PM 1/6/2006 +0100, Matthias Urlichs wrote: >Hi, > >Phillip J. Eby: > > Um, the "other kind" I'm referring to here are the ones that are > > exclusively part of the extension in question, e.g. libPyICU. This > library > > is not going to be found anywhere else except in other versions of PyICU, > > so that issue doesn't really apply here, since only one egg will have > > extensions loading the library into a given process. > >So why don't you simply link the Python extension and the library >to one single .so file? Because there are a half-dozen Python extensions in the project sharing the library. Hence, the need for a "shared library". :) From smurf at smurf.noris.de Fri Jan 6 18:57:00 2006 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Fri, 6 Jan 2006 18:57:00 +0100 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060106115725.02014b18@mail.telecommunity.com> References: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060106115725.02014b18@mail.telecommunity.com> Message-ID: <20060106175700.GI17399@kiste.smurf.noris.de> Hi, Phillip J. Eby: > >So why don't you simply link the Python extension and the library > >to one single .so file? > > Because there are a half-dozen Python extensions in the project sharing the > library. Hence, the need for a "shared library". :) > So put them all into one .so file and import the required parts from the modules you want them in. Anyway, ELF libraries do have an rpath embedded in them. It *is* possible to hotpatch that to point at wherever the shared library ends up being installed. Debian has a package "chrpath" with a tool "chrpath" (what a surprise ;-) which can do that. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - "I daresay," said Granny, pushing the Fool aside and stepping over a writhing taproot. "If anyone locked *me* in a dungeon, there'd be screams." -- Terry Pratchett (Wyrd Sisters) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060106/d870545a/attachment.pgp From ben at groovie.org Fri Jan 6 22:55:54 2006 From: ben at groovie.org (Ben Bangert) Date: Fri, 6 Jan 2006 13:55:54 -0800 Subject: [Distutils] Automatically going for a pre-req dev Message-ID: <8DDA1661-0B47-4DD0-B087-968670C78C28@groovie.org> When installing a package that requires the dev version of another package, it fails to to and download a dev version of the package. This appears to be because it either searches the filesystem for the prereq version, or asks CheeseShop. Given that the prereq indicates it wants a dev version, it would seem reasonable to search CheeseShop with ==dev equivalent active. As an example, if you install: easy_install -U hellahella==dev It installs fine from svn, then goes back searching for Pylons, and gets: Processing dependencies for hellahella==0.1dev-r576 Searching for Pylons>=0.1dev-r289 Reading http://www.python.org/pypi/Pylons/ Reading http://pylons.groovie.org/ No local packages or download links found for Pylons>=0.1dev-r289 error: Could not find distribution for Requirement.parse ('Pylons>=0.1dev-r289') Yet, if you run: easy_install -U Pylons==dev It has no problem fetching the latest Pylons and the Paste prereqs (which would have the same problem if I hadn't linked to them explicitly on the Pylons page). Since you already indicate you want ==dev, shouldn't it keep that during the prereq if necessary? - Ben From pje at telecommunity.com Fri Jan 6 23:48:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Jan 2006 17:48:07 -0500 Subject: [Distutils] Automatically going for a pre-req dev In-Reply-To: <8DDA1661-0B47-4DD0-B087-968670C78C28@groovie.org> Message-ID: <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> At 01:55 PM 1/6/2006 -0800, Ben Bangert wrote: >When installing a package that requires the dev version of another >package, it fails to to and download a dev version of the package. >This appears to be because it either searches the filesystem for the >prereq version, or asks CheeseShop. > >Given that the prereq indicates it wants a dev version, it would seem >reasonable to search CheeseShop with ==dev equivalent active. > >As an example, if you install: >easy_install -U hellahella==dev > >It installs fine from svn, then goes back searching for Pylons, and >gets: >Processing dependencies for hellahella==0.1dev-r576 >Searching for Pylons>=0.1dev-r289 >Reading http://www.python.org/pypi/Pylons/ >Reading http://pylons.groovie.org/ >No local packages or download links found for Pylons>=0.1dev-r289 >error: Could not find distribution for Requirement.parse >('Pylons>=0.1dev-r289') > >Yet, if you run: >easy_install -U Pylons==dev > >It has no problem fetching the latest Pylons and the Paste prereqs >(which would have the same problem if I hadn't linked to them >explicitly on the Pylons page). > >Since you already indicate you want ==dev, shouldn't it keep that >during the prereq if necessary? There is nothing special about 'dev' as a version; it could be "Pylons=any_old_version" for all easy_install knows. What you want is to set your install_requires=['Pylons==dev,>=0.1dev-r289'] instead. This tells easy_install that if it sees a link to a 'dev' version, go ahead and install it, but at runtime the distribution will have to match the other version, because there is really no "dev" version. Which is why the trick works, basically. If you ask for ==dev *or* >=0.1dev-r289, the latter will not be found on the Cheeseshop (until an 0.1 final is released), but a link to the "dev" version will. At runtime, nothing matches "dev", but it *will* match the 0.1dev. Anyway, this trick could be played with any special version in place of "dev", but that is currently the convention being used by most people posting links to SVN versions. It's the responsibility of the person defining the dependencies to say whether they want development versions of their requirements. From ben at groovie.org Sat Jan 7 02:08:09 2006 From: ben at groovie.org (Ben Bangert) Date: Fri, 6 Jan 2006 17:08:09 -0800 Subject: [Distutils] Automatically going for a pre-req dev In-Reply-To: <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> References: <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> Message-ID: On Jan 6, 2006, at 2:48 PM, Phillip J. Eby wrote: > There is nothing special about 'dev' as a version; it could be > "Pylons=any_old_version" for all easy_install knows. > > What you want is to set your install_requires= > ['Pylons==dev,>=0.1dev-r289'] instead. This tells easy_install > that if it sees a link to a 'dev' version, go ahead and install it, > but at runtime the distribution will have to match the other > version, because there is really no "dev" version. > > Which is why the trick works, basically. If you ask for ==dev *or* > >=0.1dev-r289, the latter will not be found on the Cheeseshop > (until an 0.1 final is released), but a link to the "dev" version > will. At runtime, nothing matches "dev", but it *will* match the > 0.1dev. > > Anyway, this trick could be played with any special version in > place of "dev", but that is currently the convention being used by > most people posting links to SVN versions. It's the responsibility > of the person defining the dependencies to say whether they want > development versions of their requirements. Excellent! That did the trick perfectly. :) Btw, off topic for this thread, but I didn't see an earlier message of mine go through regarding python-openid not installing, it gets this error: ns# easy_install -U python-openid Searching for python-openid Reading http://www.python.org/pypi/python-openid/ Reading http://www.openidenabled.com/openid/libraries/python/ Reading http://www.openidenabled.com/openid/libraries/python/ downloads/python-openid-1-0-3-tar.gz/download No local packages or download links found for python-openid error: Could not find distribution for Requirement.parse('python- openid') Apparently it doesn't like Plone download links? Thanks, Ben From pje at telecommunity.com Sat Jan 7 02:18:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Jan 2006 20:18:32 -0500 Subject: [Distutils] Automatically going for a pre-req dev In-Reply-To: References: <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060106201130.02263d10@mail.telecommunity.com> At 05:08 PM 1/6/2006 -0800, Ben Bangert wrote: >ns# easy_install -U python-openid >Searching for python-openid >Reading http://www.python.org/pypi/python-openid/ >Reading http://www.openidenabled.com/openid/libraries/python/ >Reading http://www.openidenabled.com/openid/libraries/python/ >downloads/python-openid-1-0-3-tar.gz/download >No local packages or download links found for python-openid >error: Could not find distribution for Requirement.parse('python- openid') > >Apparently it doesn't like Plone download links? Yep. Not very RESTful, that. :) The second problem is that the filename isn't a valid distutils source distro filename for a '1.0.3' version; the name should be python-openid-1.0.3.tar.gz instead. You can work around both of these issues by adding '#egg=python-openid-1.0.3' to the end of the download URL on your PyPI page, which explicitly tells easy_install how to interpret the link. Note, however, that #egg link fragments are intended mainly for Subversion checkouts, and easy_install will therefore give a higher match precedence to any binary links that match, *including* any .tar.gz or other archive links that don't use #egg. (Unless the user requests an --editable checkout, in which case #egg links and source distributions have the highest priority.) Anyway, it's probably better in the long run to simply make files available with the standard distutils filename, and use #egg in this case only as a workaround. From kevin at janrain.com Sat Jan 7 03:16:36 2006 From: kevin at janrain.com (Kevin Turner) Date: Fri, 06 Jan 2006 18:16:36 -0800 Subject: [Distutils] funny looking download links (was: Automatically going for a pre-req dev) In-Reply-To: <5.1.1.6.0.20060106201130.02263d10@mail.telecommunity.com> References: <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> <5.1.1.6.0.20060106201130.02263d10@mail.telecommunity.com> Message-ID: <1136600197.25623.28.camel@troglodyte.asianpear> On Fri, 2006-01-06 at 20:18 -0500, Phillip J. Eby wrote: > At 05:08 PM 1/6/2006 -0800, Ben Bangert wrote: > >ns# easy_install -U python-openid > >Searching for python-openid > >Reading http://www.python.org/pypi/python-openid/ > >Reading http://www.openidenabled.com/openid/libraries/python/ > >Reading http://www.openidenabled.com/openid/libraries/python/ > downloads/python-openid-1-0-3-tar.gz/download > >No local packages or download links found for python-openid > >error: Could not find distribution for Requirement.parse('python- openid') > > > >Apparently it doesn't like Plone download links? > > Yep. Not very RESTful, that. :) The second problem is that the filename > isn't a valid distutils source distro filename for a '1.0.3' version; the > name should be python-openid-1.0.3.tar.gz instead. Huh? It's RESTful, you're just overly concerned with the syntax of the URL. But okay, we'll serve the files with a less ridiculous web server. First though, let me play devil's advocate for a moment here: I entered this URL into the "download URL" field of a record that explicitly refers to the 1.0.3 version of this package. Why not use it? If you're worried about ending up with a file named "download" that you don't know how to unpack, the reasonable name (python-openid-1.0.3.tar.gz) is specified in the not-really-standard-but-well-documented Content-Disposition HTTP header. (I'd bug the Zope guys about this, but I can see how it'd probably break their object publishing model to do it differently. If I were lucky, they might give me http://www.openidenabled.com/openid/libraries/python/downloads/python-openid-1-0-3-tar.gz/download/python-openid-1.0.3.tar.gz , and then I'd have to patch cheeseshop so that the overly long URL wouldn't hose the page formatting any more than it already does...) -- The moon is first quarter, 49.6% illuminated, 7.3 days old. From pje at telecommunity.com Sat Jan 7 03:53:04 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Jan 2006 21:53:04 -0500 Subject: [Distutils] funny looking download links (was: Automatically going for a pre-req dev) In-Reply-To: <1136600197.25623.28.camel@troglodyte.asianpear> References: <5.1.1.6.0.20060106201130.02263d10@mail.telecommunity.com> <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> <5.1.1.6.0.20060106174211.02238600@mail.telecommunity.com> <5.1.1.6.0.20060106201130.02263d10@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060106213815.02266490@mail.telecommunity.com> At 06:16 PM 1/6/2006 -0800, Kevin Turner wrote: >On Fri, 2006-01-06 at 20:18 -0500, Phillip J. Eby wrote: > > At 05:08 PM 1/6/2006 -0800, Ben Bangert wrote: > > >ns# easy_install -U python-openid > > >Searching for python-openid > > >Reading http://www.python.org/pypi/python-openid/ > > >Reading http://www.openidenabled.com/openid/libraries/python/ > > >Reading http://www.openidenabled.com/openid/libraries/python/ > > downloads/python-openid-1-0-3-tar.gz/download > > >No local packages or download links found for python-openid > > >error: Could not find distribution for Requirement.parse('python- openid') > > > > > >Apparently it doesn't like Plone download links? > > > > Yep. Not very RESTful, that. :) The second problem is that the filename > > isn't a valid distutils source distro filename for a '1.0.3' version; the > > name should be python-openid-1.0.3.tar.gz instead. > >Huh? It's RESTful, you're just overly concerned with the syntax of the >URL. My point (such as it is) was that REST prefers to avoid putting verbs (e.g. download) in URLs. There was also a ":)" to indicate that it was an attempt at humor. :) >First though, let me play devil's advocate for a moment here: I entered >this URL into the "download URL" field of a record that explicitly >refers to the 1.0.3 version of this package. Why not use it? 1. Because a significant % of the packages on PyPI have a download URL that points to an HTML page, which then (sometimes) has links to the actual downloadable packages. 2. Because easy_install needs to know what version of the package it's dealing with and what format, in order to select the appropriate package. Since easy_install can collect data from various web pages in addition to what's on PyPI (as well as an arbitrary number of links found on a PyPI page), it needs a parseable filename. >If you're worried about ending up with a file named "download" that you >don't know how to unpack, the reasonable name >(python-openid-1.0.3.tar.gz) is specified in the >not-really-standard-but-well-documented Content-Disposition HTTP header. The irony is that easy_install *is* downloading the file, but then ignores it because it's not an HTML page. I suppose I could get it to sniff the content type and disposition, but in my experience so many servers have differently configured values for the content-type of .tar.gz files that there isn't anything particularly stable about content-type. I suppose if the tool encounters a non-HTML page, I could have it try to interpret the suggested filename and pretend it got that information from the URL. This would result in two downloads of the same file, of course, but if you want to avoid that, you can just use the #egg trick. Anyway, even assuming I actually implement this, it'll take a while for everybody to update their setuptools such that they could actually use it. In other words, you'll need the #egg trick for some time to come no matter what. >(I'd bug the Zope guys about this, but I can see how it'd probably break >their object publishing model to do it differently. If I were lucky, >they might give me >http://www.openidenabled.com/openid/libraries/python/downloads/python-openid-1-0-3-tar.gz/download/python-openid-1.0.3.tar.gz >, and then I'd have to patch cheeseshop so that the overly long URL >wouldn't hose the page formatting any more than it already does...) Just include the URL in your long_description field, and if you use reST (reStructured Text) you can give it a nice human-readable content that doesn't display the URL. For an example, see http://cheeseshop.python.org/pypi/setuptools which has several links in the long_description, one of which is an #egg link for the SVN version. From stephen.langer at earthlink.net Sat Jan 7 03:55:04 2006 From: stephen.langer at earthlink.net (Stephen Langer) Date: Fri, 6 Jan 2006 21:55:04 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: References: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060106111637.0203ca00@mail.telecommunity.com> Message-ID: <88BEAFC0-0C16-42FB-85C4-E17CA39D28F4@earthlink.net> On Jan 6, 2006, at 11:33 AM, Bob Ippolito wrote: > > On Jan 6, 2006, at 8:19 AM, Phillip J. Eby wrote: >> >> The issue for the PyICU/PyLucene etc. use cases is that there are a >> whole >> bunch of SWIG wrappers for different extensions, but with a large >> common >> library for type mapping etc. The authors want a shared library >> for *their >> own code* that is used by multiple extensions. This is not >> separately >> distributed code. > > It'd be better if these projects just fixed themselves to do things > the "right" way, using a Python extension and a function table. Bob -- Can you explain to me in more detail what the "right" way is? I have a large number of swig wrapper files for a large number of C++ files. The program has a small number of core shared libraries that are always loaded, and a few that are optional. The shared library code is written in C++. Sometimes classes defined in the core libraries are used as base classes for classes defined in the optional libraries. All of this can be handled easily with shared libraries. How can it be done with function tables? Thanks, Steve -- -- EMail: stephen.langer at nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 -- From bob at redivi.com Sat Jan 7 04:27:56 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 6 Jan 2006 19:27:56 -0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <88BEAFC0-0C16-42FB-85C4-E17CA39D28F4@earthlink.net> References: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060106111637.0203ca00@mail.telecommunity.com> <88BEAFC0-0C16-42FB-85C4-E17CA39D28F4@earthlink.net> Message-ID: On Jan 6, 2006, at 6:55 PM, Stephen Langer wrote: > > On Jan 6, 2006, at 11:33 AM, Bob Ippolito wrote: > >> >> On Jan 6, 2006, at 8:19 AM, Phillip J. Eby wrote: >>> >>> The issue for the PyICU/PyLucene etc. use cases is that there are a >>> whole >>> bunch of SWIG wrappers for different extensions, but with a large >>> common >>> library for type mapping etc. The authors want a shared library >>> for *their >>> own code* that is used by multiple extensions. This is not >>> separately >>> distributed code. >> >> It'd be better if these projects just fixed themselves to do things >> the "right" way, using a Python extension and a function table. > > Can you explain to me in more detail what the "right" way is? I > have a large number of swig wrapper files for a large number of C++ > files. The program has a small number of core shared libraries > that are always loaded, and a few that are optional. The shared > library code is written in C++. Sometimes classes defined in the > core libraries are used as base classes for classes defined in the > optional libraries. All of this can be handled easily with shared > libraries. How can it be done with function tables? Numeric, Numarray, pygame, PyObjC, etc. are good references for how to export bits of C API as a function table. I don't use C++ unless I have to, so I'm not familiar enough with it to say whether or not it's going to cause problems given how you want to use it. Maybe it's not possible without refactoring the optional library code to work some other way. -bob From jeff.pitman at gmail.com Sat Jan 7 05:59:06 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Sat, 7 Jan 2006 12:59:06 +0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <20060106052014.GZ17399@kiste.smurf.noris.de> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> Message-ID: <6b9c17630601062059p74249628p2e44ee3955841c4d@mail.gmail.com> On 1/7/06, Phillip J. Eby wrote: > At 06:20 AM 1/6/2006 +0100, Matthias Urlichs wrote: > >Besides, Unix systems these days tend to have the required libraries > >installed anyway. > > Not libPyICU, since it's a part of PyICU. Similar issues may apply to > other multi-extension projects that need to share a large chunk of code. Whether this is the right way to do things is up for debate. Earlier in the thread libpng.so was thrown out as example--this immediately threw some red flags up because it's a common/standard lib that comes with most distros. If you need .so that's built and exported by the same packaged egg, you may just consider installing it in a more benign area such as /usr/local/lib. Though, the *BSD folks may also have some thoughts on the matter. Messing with LD_LIBRARY_PATH is never the right way to go about solving this. Unless you're Oracle... Please keep in mind that the different *nixes will want to stick with their own package formats, managers, and automatic dep solvers. Enhancing setuptools to facilitate these (bdist_rpm/bdist_deb), will be more beneficial in the long run. -- -jeff From pje at telecommunity.com Sat Jan 7 06:55:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 07 Jan 2006 00:55:32 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <6b9c17630601062059p74249628p2e44ee3955841c4d@mail.gmail.co m> References: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <20060106052014.GZ17399@kiste.smurf.noris.de> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060107001420.02009ee8@mail.telecommunity.com> At 12:59 PM 1/7/2006 +0800, Jeff Pitman wrote: > If you need .so that's built and exported by the same packaged egg, you > may just consider installing it in a more benign area such as /usr/local/lib. As I mentioned before, this breaks multi-version support, although I suppose one could perhaps force the egg version to always be included in the filename... that would lead to some interesting problems as far as keeping the build area clean, though! You'd have to run a "clean --all" whenever the version number changed, and the extensions using the library would have to refer to the exact version. Unfortunately, this would *still* require users to mess with LD_LIBRARY_PATH in order to use private library versions, and so isn't particularly useful. Some variant of the "patch the extensions' runtime path at installation/cache extraction time" approach would still be necessary. There is actually a hook in pkg_resources whose purpose was to support doing this kind of thing with any code resources extracted to the egg cache, so it's probably doable, and I could probably also make the code usable by easy_install during egg unpacking. It pretty much means the "unzip the egg manually" concept is a dead duck when libraries are involved, though. You could only safely unpack an egg using a manually-created ResourceManager with an appropriate extraction path base. On Mac OS there's an option to automatically pad the rpath area, but on other OSes I'd need to do the padding myself by supplying an extra-long rpath entry that could then be overwritten during the extraction process. What bothers me about all of this is that moving an installed egg then *breaks* it, because the rpath is now hard-wired. Sigh. Ironically enough, an egg containing a shared library would then need to be labeled as "not unzip safe", i.e., it's really only safe to use if it remains zipped. A new category of egg, to be sure. :) Interestingly, this is one of the few times where I have to say that Windows is actually better-designed than Unix. Sure, the same design choice enables all other sorts of DLL hell, but on this very narrow specific issue Windows kicks ass. :) Anyway, I really wish there were a way to hook into the dynamic link process that didn't require one to manage the entire linkage one's self. The dl.open() function lets you get access to symbols, but that doesn't seem sufficient. :( From jeff.pitman at gmail.com Sat Jan 7 07:01:14 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Sat, 7 Jan 2006 14:01:14 +0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060107001420.02009ee8@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <20060106052014.GZ17399@kiste.smurf.noris.de> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060107001420.02009ee8@mail.telecommunity.com> Message-ID: <6b9c17630601062201k498aa2d6y68d64e9f5ae32017@mail.gmail.com> On 1/7/06, Phillip J. Eby wrote: > Interestingly, this is one of the few times where I have to say that > Windows is actually better-designed than Unix. Sure, the same design > choice enables all other sorts of DLL hell, but on this very narrow > specific issue Windows kicks ass. :) There is an alternative: don't worry about it. It's just not going to be used that often to justify putting in a whole slew of hackery to patch a package that's justifyingly doing something wrong in the first place. -- -jeff From bob at redivi.com Sat Jan 7 07:20:48 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 6 Jan 2006 22:20:48 -0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060107001420.02009ee8@mail.telecommunity.com> References: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <20060106052014.GZ17399@kiste.smurf.noris.de> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060107001420.02009ee8@mail.telecommunity.com> Message-ID: <211442D7-17D9-4ED9-9309-FB5C3A903624@redivi.com> On Jan 6, 2006, at 9:55 PM, Phillip J. Eby wrote: > At 12:59 PM 1/7/2006 +0800, Jeff Pitman wrote: >> If you need .so that's built and exported by the same packaged >> egg, you >> may just consider installing it in a more benign area such as /usr/ >> local/lib. > > As I mentioned before, this breaks multi-version support, although I > suppose one could perhaps force the egg version to always be > included in > the filename... that would lead to some interesting problems as > far as > keeping the build area clean, though! You'd have to run a "clean -- > all" > whenever the version number changed, and the extensions using the > library > would have to refer to the exact version. > > Unfortunately, this would *still* require users to mess with > LD_LIBRARY_PATH in order to use private library versions, and so isn't > particularly useful. Some variant of the "patch the extensions' > runtime > path at installation/cache extraction time" approach would still be > necessary. > > There is actually a hook in pkg_resources whose purpose was to support > doing this kind of thing with any code resources extracted to the egg > cache, so it's probably doable, and I could probably also make the > code > usable by easy_install during egg unpacking. It pretty much means the > "unzip the egg manually" concept is a dead duck when libraries are > involved, though. You could only safely unpack an egg using a > manually-created ResourceManager with an appropriate extraction > path base. > > On Mac OS there's an option to automatically pad the rpath area, > but on > other OSes I'd need to do the padding myself by supplying an extra- > long > rpath entry that could then be overwritten during the extraction > process. IIRC, you can't set that any DYLD environment variables after the process has started... has to happen before main() is even called. > What bothers me about all of this is that moving an installed egg then > *breaks* it, because the rpath is now hard-wired. Sigh. Ironically > enough, an egg containing a shared library would then need to be > labeled as > "not unzip safe", i.e., it's really only safe to use if it remains > zipped. A new category of egg, to be sure. :) > > Interestingly, this is one of the few times where I have to say that > Windows is actually better-designed than Unix. Sure, the same design > choice enables all other sorts of DLL hell, but on this very narrow > specific issue Windows kicks ass. :) Well, it's not really the design that's better, just the defaults... for a questionable definition of better :) -bob From pje at telecommunity.com Sat Jan 7 17:20:40 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 07 Jan 2006 11:20:40 -0500 Subject: [Distutils] Dynamic linking - another idea Message-ID: <5.1.1.6.0.20060107110522.02009ed8@mail.telecommunity.com> Okay, so here's a crazy idea to make dynamic libraries in the same package work. Suppose the linker includes an rpath of '.' (aka os.curdir), and renames each extension module that's using a library such that it's not directly importable by Python. Setuptools already generates a stub .py file loader for extension modules anyway during bdist_egg, so that zipped extensions can be extracted. So, this stub could just change directory to the directory of the extension before doing the import, with setdlopenflags(RTLD_NOW). If I understand correctly, this should force all symbols in the extension to be resolved immediately, so the '.' rpath would let the extension load the nearby library. After this was done, the stub code would restore the original current directory and dlopen flags. Since extensions built this way would not be directly importable, it would ensure that the stub is always used. And since they do not require a hardcoded real path, they would be relocatable and manually extractable from the zipfile. Does any see any holes in this, aside from the sheer craziness of doing it? :) That is, are there any platforms where a '.' rpath is forbidden? Where RTLD_NOW won't let it resolve all the symbols? Where the '.' rpath is interpreted as the executable's startup directory rather than as the load-time current directory? From pje at telecommunity.com Tue Jan 10 05:14:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 09 Jan 2006 23:14:23 -0500 Subject: [Distutils] New features in setuptools SVN Message-ID: <5.1.1.6.0.20060109230318.02237850@mail.telecommunity.com> I just checked in a couple of new features for EasyInstall that some of you will be interested in. First, '.' is now allowed in project names, and '_' and '-' in names should no longer confuse EasyInstall when it searches PyPI. It will use whatever characters you originally entered. The bad news is that this could break any existing dependencies to projects with a '.' in their names, because the '.' was being converted to '-' before, and isn't now. The good news is that there are only 5 PyPI projects with a '.' in the name (rsync.py, spider.py, _sre.py, zope.interface, and zope.testbrowser), and of them only '_sre.py' had a prayer of working with EasyInstall before. Now, 'spider.py' also works due to the next new feature, which is that installation of bare '.py' files (no accompanying setup.py) is now supported. If a PyPI "Download URL" points to a file with a .py extension, EasyInstall will automatically create a proper setup.py alongside it. It can also do this for any link to a .py file that includes an '#egg=name-version' fragment identifier, so that you can use a link like this: http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py#egg=PySourceColor-2.1a_20050425 to point to somebody's .py file and assign it a project name and version. Note that the '#egg' tag must be *unambiguous*, which means you cannot have a '-' in the project name or version. You must escape them as '_' characters, using the '-' only to delimit the name from the version. Anyway, if you do this on *your* project's PyPI page to link to your project's unpackaged dependencies, you can fill in for the dependency's lack of a PyPI page or setup script, while still using the normal setuptools way of specifying project dependencies. Please let me know if you have any questions or problems using these new features. Thanks! From trentm at ActiveState.com Tue Jan 10 18:32:42 2006 From: trentm at ActiveState.com (Trent Mick) Date: Tue, 10 Jan 2006 09:32:42 -0800 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060107001420.02009ee8@mail.telecommunity.com> References: <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <20060106052014.GZ17399@kiste.smurf.noris.de> <5.1.1.6.0.20060106113249.03a476c0@mail.telecommunity.com> <5.1.1.6.0.20060107001420.02009ee8@mail.telecommunity.com> Message-ID: <20060110173242.GA32199@activestate.com> [Phillip J. Eby wrote] > ... > Some variant of the "patch the extensions' runtime path at > installation/cache extraction time" approach would still be necessary. Yup, unfortunately. > There is actually a hook in pkg_resources whose purpose was to support > doing this kind of thing with any code resources extracted to the egg > cache, so it's probably doable, and I could probably also make the code > usable by easy_install during egg unpacking. It pretty much means the > "unzip the egg manually" concept is a dead duck when libraries are > involved, though. You could only safely unpack an egg using a > manually-created ResourceManager with an appropriate extraction path base. Perhaps 'bin/egg-unzip' or something like that? > On Mac OS there's an option to automatically pad the rpath area, but on > other OSes I'd need to do the padding myself by supplying an extra-long > rpath entry that could then be overwritten during the extraction process. Yes, that is what I do for ActivePython (and similar for Active{Tcl|Perl}) for the Unix OSes. Effectively: configure --prefix=.../some_really_really_long_path make && make install > What bothers me about all of this is that moving an installed egg then > *breaks* it, because the rpath is now hard-wired. Sigh. Ironically > enough, an egg containing a shared library would then need to be labeled as > "not unzip safe", i.e., it's really only safe to use if it remains But still "egg-unzip" safe, right? :) Aside: I've found this page to be quite helpful for cross-platform shared library handling info: http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html Cheers, Trent -- Trent Mick trentm at activestate.com From ianb at colorstudy.com Wed Jan 11 03:31:01 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 10 Jan 2006 20:31:01 -0600 Subject: [Distutils] setuptools: confusion with non-egg installed packages Message-ID: <43C46DE5.1000306@colorstudy.com> I had a confusing installation experience today. Trac was installed in /usr/local/lib/python2.4/site-packages/trac, and I checked out a version that I had hacked to use setuptools, and run "python setup.py develop" on. This added my checkout to the path, but after site-packages, so it wouldn't be loaded, even if I did require('trac'). Now, if I did setup.py develop -m it would actually work, because setuptools would fix up the path correctly. I don't know the best resolution here. Maybe develop should check for conflicting installed packages? -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From pje at telecommunity.com Wed Jan 11 06:26:13 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 11 Jan 2006 00:26:13 -0500 Subject: [Distutils] setuptools: confusion with non-egg installed packages In-Reply-To: <43C46DE5.1000306@colorstudy.com> Message-ID: <5.1.1.6.0.20060111002507.01fb25d8@mail.telecommunity.com> At 08:31 PM 1/10/2006 -0600, Ian Bicking wrote: >I don't know the best resolution here. Maybe develop should check for >conflicting installed packages? I'm kind of surprised it doesn't. Added to my to-do list for 0.6a10. From stephen.langer at nist.gov Wed Jan 11 21:32:30 2006 From: stephen.langer at nist.gov (Stephen Langer) Date: Wed, 11 Jan 2006 15:32:30 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: References: <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105235135.0203e4f0@mail.telecommunity.com> <5.1.1.6.0.20060106111637.0203ca00@mail.telecommunity.com> <88BEAFC0-0C16-42FB-85C4-E17CA39D28F4@earthlink.net> Message-ID: On Jan 6, 2006, at 10:27 PM, Bob Ippolito wrote: > On Jan 6, 2006, at 6:55 PM, Stephen Langer wrote: >> On Jan 6, 2006, at 11:33 AM, Bob Ippolito wrote: >>> On Jan 6, 2006, at 8:19 AM, Phillip J. Eby wrote: >>>> >>>> The issue for the PyICU/PyLucene etc. use cases is that there >>>> are a whole >>>> bunch of SWIG wrappers for different extensions, but with a >>>> large common >>>> library for type mapping etc. The authors want a shared library >>>> for *their >>>> own code* that is used by multiple extensions. This is not >>>> separately >>>> distributed code. >>> >>> It'd be better if these projects just fixed themselves to do things >>> the "right" way, using a Python extension and a function table. >> >> Can you explain to me in more detail what the "right" way is? I >> have a large number of swig wrapper files for a large number of C+ >> + files. The program has a small number of core shared libraries >> that are always loaded, and a few that are optional. The shared >> library code is written in C++. Sometimes classes defined in the >> core libraries are used as base classes for classes defined in the >> optional libraries. All of this can be handled easily with shared >> libraries. How can it be done with function tables? > > Numeric, Numarray, pygame, PyObjC, etc. are good references for how > to export bits of C API as a function table. I don't use C++ > unless I have to, so I'm not familiar enough with it to say whether > or not it's going to cause problems given how you want to use it. > Maybe it's not possible without refactoring the optional library > code to work some other way. Ok, I've looked at some of these examples. I haven't had the time to go through their mechanisms in detail, but it seems that under the hood they don't do anything terribly different from what swig does. I still don't see how they pertain to solving the shared library problem, and how doing things the "right" way makes shared libraries unnecessary. If extension A and extension B both require routine Z, and there's no python extension for routine Z, isn't the best solution to put Z in a shared library? I hope I'm not being dense here. We have lots of python extensions, all of which link to a relatively small number of shared libraries, and I'd like to understand why this setup is "wrong". -- Steve -- -- EMail: stephen.langer at nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 -- From strawman at astraw.com Thu Jan 12 02:31:58 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed, 11 Jan 2006 17:31:58 -0800 Subject: [Distutils] making easy_install install extras Message-ID: <43C5B18E.5090108@astraw.com> Hi, Is it possible to convince easy_install to grab extras specified in the "extras_require" arguments to setup()? I know this probably isn't something that should be on by default (that's why they're "extras"), but it would be nice to do something like: easy_install some_package --also-install-all-available-extras If it's not currently possible, I'd like to submit this as a feature request. Cheers! Andrew From pje at telecommunity.com Thu Jan 12 03:12:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 11 Jan 2006 21:12:41 -0500 Subject: [Distutils] making easy_install install extras In-Reply-To: <43C5B18E.5090108@astraw.com> Message-ID: <5.1.1.6.0.20060111211147.01fd9b98@mail.telecommunity.com> At 05:31 PM 1/11/2006 -0800, Andrew Straw wrote: >Is it possible to convince easy_install to grab extras specified in the >"extras_require" arguments to setup()? I know this probably isn't >something that should be on by default (that's why they're "extras"), >but it would be nice to do something like: > >easy_install some_package --also-install-all-available-extras > >If it's not currently possible, I'd like to submit this as a feature >request. The syntax that should currently work is: easy_install "some_package[extra1,extra2]" There isn't a way to install an arbitrary collection of extras, however. From pje at telecommunity.com Thu Jan 12 03:17:17 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 11 Jan 2006 21:17:17 -0500 Subject: [Distutils] making easy_install install extras In-Reply-To: <5.1.1.6.0.20060111211147.01fd9b98@mail.telecommunity.com> References: <43C5B18E.5090108@astraw.com> Message-ID: <5.1.1.6.0.20060111211548.03ac6b80@mail.telecommunity.com> At 09:12 PM 1/11/2006 -0500, Phillip J. Eby wrote: >There isn't a way to install an arbitrary collection of extras, however. Eep. What I meant to say was, you can't just "install all the extras" for some arbitrary collection of extras. You *can* install an arbitrary collection of extras as long as you explicitly list all of them. :) From cduffy at spamcop.net Thu Jan 12 18:17:45 2006 From: cduffy at spamcop.net (Charles Duffy) Date: Thu, 12 Jan 2006 11:17:45 -0600 Subject: [Distutils] man pages (& gzip & bdist_rpm on SuSE, oh my!) Message-ID: I'm trying to use bdist_rpm to build RPMs for a number of 3rd-party packages on SuSE SLES9. Among the issues I've run into: SuSE has a different %mandir from Red Hat -- man pages go under /usr/share/man rather than /usr/man. Upstream setup.py specifies data_files=[('man/man1', ['bzr.1'])]. I can change this to reference 'share/man/man1', which lets me build my package -- but it'd be nice to have something I can distribute upstream. Do I need to actually look at the filesystem and determine whether /usr/share/man exists and pass in the appropriate path, or is there a better approach? In /usr/lib/rpm/suse-macros, an __os_install_post macro is specified which calls %{suse_check} and then /usr/lib/rpm/brp-compress. Either of these will compress man pages (among other files), should it find them. This leads to the issue described in http://mail.python.org/pipermail/distutils-sig/2001-December/002674.html. Other than temporarily modifying /usr/lib/rpm/suse-macros, are there any ready workarounds? Thanks! From pje at telecommunity.com Thu Jan 12 19:18:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 12 Jan 2006 13:18:05 -0500 Subject: [Distutils] making easy_install install extras In-Reply-To: <43C69B23.8080509@astraw.com> References: <5.1.1.6.0.20060111211147.01fd9b98@mail.telecommunity.com> <5.1.1.6.0.20060111211147.01fd9b98@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060112131606.01e4bf70@mail.telecommunity.com> At 10:08 AM 1/12/2006 -0800, Andrew Straw wrote: >Phillip J. Eby wrote: > > > At 05:31 PM 1/11/2006 -0800, Andrew Straw wrote: > > > >> Is it possible to convince easy_install to grab extras specified in the > >> "extras_require" arguments to setup()? I know this probably isn't > >> something that should be on by default (that's why they're "extras"), > >> but it would be nice to do something like: > >> > >> easy_install some_package --also-install-all-available-extras > >> > >> If it's not currently possible, I'd like to submit this as a feature > >> request. > > > > > > The syntax that should currently work is: > > > > easy_install "some_package[extra1,extra2]" > >This is fine (if some extra typing) if you know what those extras are. >Is there any way to ask easy_install to display what extras some_package >can use? I'll expand the "package successfully installed" message to list the extras and give an example of how to activate them. This is something project authors should include in their documentation, though, since it may not be obvious what a given extra "foo" does, and there isn't presently any place to describe that in the metadata, either. From strawman at astraw.com Thu Jan 12 19:08:35 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 12 Jan 2006 10:08:35 -0800 Subject: [Distutils] making easy_install install extras In-Reply-To: <5.1.1.6.0.20060111211147.01fd9b98@mail.telecommunity.com> References: <5.1.1.6.0.20060111211147.01fd9b98@mail.telecommunity.com> Message-ID: <43C69B23.8080509@astraw.com> Phillip J. Eby wrote: > At 05:31 PM 1/11/2006 -0800, Andrew Straw wrote: > >> Is it possible to convince easy_install to grab extras specified in the >> "extras_require" arguments to setup()? I know this probably isn't >> something that should be on by default (that's why they're "extras"), >> but it would be nice to do something like: >> >> easy_install some_package --also-install-all-available-extras >> >> If it's not currently possible, I'd like to submit this as a feature >> request. > > > The syntax that should currently work is: > > easy_install "some_package[extra1,extra2]" This is fine (if some extra typing) if you know what those extras are. Is there any way to ask easy_install to display what extras some_package can use? Currently, I use easy_install to install some_package, keep typing "some_package_script_requiring_extras" to see the repeated DistributionNotFound errors and get them. It would be nice to short-circuit this process from the beginning. Cheers! Andrew From ianb at colorstudy.com Fri Jan 13 18:20:40 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 13 Jan 2006 11:20:40 -0600 Subject: [Distutils] setuptools: branch versions Message-ID: <43C7E168.7020306@colorstudy.com> One thing that would be nice, but seems rather substantial, would be a way to tag branches. Right now 0.1dev and 0.1wsgi are ordered in some way, and for branches I'd actually like for them not to be ordered -- you really want one tag or the other, and they aren't interchangeable in any way. Which maybe is possible now. Though the version specifiers make it hard too, because how do you say you want 0.1wsgi-*? Ultimately I'm looking for a "tag_svn_branch = true" setting, I guess, but the version sorting makes that less useful. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Jan 13 19:07:37 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 13 Jan 2006 13:07:37 -0500 Subject: [Distutils] setuptools: branch versions In-Reply-To: <43C7E168.7020306@colorstudy.com> Message-ID: <5.1.1.6.0.20060113130628.01e66b58@mail.telecommunity.com> At 11:20 AM 1/13/2006 -0600, Ian Bicking wrote: >One thing that would be nice, but seems rather substantial, would be a >way to tag branches. Right now 0.1dev and 0.1wsgi are ordered in some >way, and for branches I'd actually like for them not to be ordered -- >you really want one tag or the other, and they aren't interchangeable in >any way. Which maybe is possible now. Though the version specifiers >make it hard too, because how do you say you want 0.1wsgi-*? > >Ultimately I'm looking for a "tag_svn_branch = true" setting, I guess, >but the version sorting makes that less useful. To designate a persistent branch, you should modify the project name instead. E.g. 'foobar_wsgi' or 'foobar_plain'. From pje at telecommunity.com Sat Jan 14 02:37:08 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 13 Jan 2006 20:37:08 -0500 Subject: [Distutils] 2 platforms down, only Mac to go... Message-ID: <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> I've got shared libraries working on Windows and Linux -- without any LD_LIBRARY_PATH nonsense, or any .so file rewriting. I have not had a chance to try this on OS X, however. If there's anybody out there who has experience wrestling with the Mac OS linker (hi, Bob! ) has a chance to try it out and maybe tweak it a little bit, I'd appreciate it. The code is in the subversion trunk now, and the source includes a tests/shlib_test subdirectory that you can change to and run "setup.py test" to build/link/test, e.g.: cd tests/shlib_test PYTHONPATH=../.. python setup.py test This just verifies that the basic linking works, i.e., whether my guesses for the linker options in build_ext.py were correct. The second test is to verify that it works from a different directory: cd .. PYTHONPATH=..:shlib_test python -c "import hello" If this works without crashing, then the dynamic linking works even from a different directory. Based on my limited guesswork, the final test is the one most likely to fail on OS X (assuming the link step succeeded, and I'm not sure it will!). Rename the test directory temporarily: mv shlib_test something_else PYTHONPATH=..:something_else python -c "import hello" mv something_else shlib_test If this works without a bit of hacking, I'll be rather surprised, but *very* pleased. If all three tests work without any changes to the source, it's probably a bug. ;) (Specifically, it probably means setuptools fell back to static linking, and thus isn't having any dynamic link problems.) The basic idea of what I'm doing here is that I try to set the runtime library search path of each extension (that depends on a shared library in the project), to include the current directory *at runtime*, then make a Python wrapper for the extension that temporarily changes the current directory while doing the import. The tricky bit -- if I understand correctly -- is that OS X has no concept of a runtime library search path, so what this probably *should* be doing is specifying something like "-dylib_file libhellolib.dylib:./libhellolib.dylib" in the link options for the 'hello' extension. What I'm not clear on is whether this is actually allowed by the linker, or if it even does what I think it does in the first place. Even if it's allowed and does the right thing, I have a sneaky suspicion that perhaps the path will be converted to an absolute path at link time, rather than leaving the './' in place, given the general reputation for deviousness possessed by the Mac OS linker. :) On the other hand, perhaps it's possible to fudge the paths in the dylib once it's built, such that the './' is in the file? Any thoughts or suggestions would be welcome. In a couple of weeks I'll have a chance to play around with this on a Mac myself, but since I'm starting from scratch (zero Mac experience whatsoever), I'm hoping maybe someone more experienced could provide some degree of guidance by then. Thanks. From micktwomey at gmail.com Sat Jan 14 14:12:23 2006 From: micktwomey at gmail.com (Michael Twomey) Date: Sat, 14 Jan 2006 13:12:23 +0000 Subject: [Distutils] 2 platforms down, only Mac to go... In-Reply-To: <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> References: <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> Message-ID: <50a522ca0601140512h1be4f529g@mail.gmail.com> Hi, (initially this was a quick "did it work for me" report, but I kept hacking at it, so it rambles on a bit, with some partial solutions, complete diff attached.) On 14/01/06, Phillip J. Eby wrote: > I've got shared libraries working on Windows and Linux -- without any > LD_LIBRARY_PATH nonsense, or any .so file rewriting. I have not had a > chance to try this on OS X, however. If there's anybody out there who has > experience wrestling with the Mac OS linker (hi, Bob! ) has a chance > to try it out and maybe tweak it a little bit, I'd appreciate it. > I just gave this a quick go using python 2.4 and 2.5 from trunk. To get the _config_vars mangling to work it needs to be setup first (you get an AttributeError on the copy() otherwise). distutils does this in one of the get_confi_var* functions, so I stuck a quick hack in to call this: Index: setuptools/command/build_ext.py =================================================================== --- setuptools/command/build_ext.py (revision 42043) +++ setuptools/command/build_ext.py (working copy) @@ -9,7 +9,9 @@ from distutils.file_util import copy_file from setuptools.extension import Library from distutils.ccompiler import new_compiler -from distutils.sysconfig import customize_compiler, _config_vars +from distutils.sysconfig import customize_compiler, get_config_var +get_config_var("LDSHARED") +from distutils.sysconfig import _config_vars from distutils import log from distutils.errors import * > If this works without a bit of hacking, I'll be rather surprised, but > *very* pleased. If all three tests work without any changes to the source, > it's probably a bug. ;) (Specifically, it probably means setuptools fell > back to static linking, and thus isn't having any dynamic link problems.) > When I ran the tests they all worked, but setuptools had indeed built a static library. Looking at the compiler output it looks like none of the LDSHARED or CCSHARED flags you set got used. Poking further I found that you are using the dl module to test for RTLD_NOW, unfortunately dl isn't present on my mac. In fact setup.py in python's source indicates that it isn't built at all: if (dl_inc is not None) and (platform not in ['atheos', 'darwin']): exts.append( Extension('dl', ['dlmodule.c']) ) I temporarily hacked my copy to always set have_rtld to True and I got errors when setup.py tried to use the flags. It looks like the compiler has to be mentioned in the LDSHARED: tmp = _config_vars.copy() try: # XXX Help! I don't have any idea whether these are right... - _config_vars['LDSHARED'] = "-dynamiclib -undefined dynamic_lookup" + _config_vars['LDSHARED'] = "gcc -Wl,-x -dynamiclib -undefined dynamic_lookup" _config_vars['CCSHARED'] = " -dynamiclib" _config_vars['SO'] = ".dylib" customize_compiler(compiler) (For reference my flags: $ python -c"import distutils.sysconfig; print distutils.sysconfig.get_config_vars('LDSHARED', 'CCSHARED', 'SO')" ['gcc -Wl,-x -bundle -undefined dynamic_lookup', '', '.so'] ) (I've used distutils' get_config_var before to pull out compiler flags and I remember having to do fun things to build up the LDSHARED across platforms. I'll see if I have any code.) Using these flags the module built fine. The otool output doesn't look quite right though (note how the build directory is mentioned in the built module, it leads to problems, see below): $ otool -L libhellolib.dylib libhellolib.dylib: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.2) $ otool -L dl-hello.so dl-hello.so: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.2) When the test tries to use the module it barfs on the use of dl in the bootstrap code (though it does import without linker errors): testHelloMsg (test_hello.HelloWorldTest) ... ERROR ====================================================================== ERROR: testHelloMsg (test_hello.HelloWorldTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/Shared/Source/svn/python/setuptools/tests/shlib_test/test_hello.py", line 5, in testHelloMsg from hello import hello File "./hello.py", line 17, in ? File "./hello.py", line 3, in __bootstrap__ ImportError: No module named dl ---------------------------------------------------------------------- Ran 1 test in 0.035s When I comment out the dl stuff in the bootstrap code things look a lot better, the module imports and runs without problems. I can move the directory to something_else and it still imports fine. The only major niggle left is the way the linker has linked in the build directory locations: $ mv build/ build.broken $ python -c "import hello" Traceback (most recent call last): File "", line 1, in ? File "./hello.py", line 17, in ? File "./hello.py", line 13, in __bootstrap__ ImportError: Failure linking new module: /Users/Shared/Source/svn/python/setuptools/tests/shlib_test/dl-hello.so: Library not loaded: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib Referenced from: /Users/Shared/Source/svn/python/setuptools/tests/shlib_test/dl-hello.so Reason: image not found > > The tricky bit -- if I understand correctly -- is that OS X has no concept > of a runtime library search path, so what this probably *should* be doing > is specifying something like "-dylib_file > libhellolib.dylib:./libhellolib.dylib" in the link options for the 'hello' > extension. What I'm not clear on is whether this is actually allowed by > the linker, or if it even does what I think it does in the first > place. Even if it's allowed and does the right thing, I have a sneaky > suspicion that perhaps the path will be converted to an absolute path at > link time, rather than leaving the './' in place, given the general > reputation for deviousness possessed by the Mac OS linker. :) > If I compile and link by hand all in the single directory then I get a fully relocatable module free of references to the build directory. It's looking like the linker is building paths relative to where it's invoked from. Re-ordering the -L flags didn't seem to help either. > On the other hand, perhaps it's possible to fudge the paths in the dylib > once it's built, such that the './' is in the file? Any thoughts or > suggestions would be welcome. > > In a couple of weeks I'll have a chance to play around with this on a Mac > myself, but since I'm starting from scratch (zero Mac experience > whatsoever), I'm hoping maybe someone more experienced could provide some > degree of guidance by then. Thanks. > I'll see if I can tweak the flags a bit, I mightn't be able to come up with anything but I thought you might like the early feedback. I'm sure someone with more OS X linking experience than me will come up with better solutions. cheers, Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: build_ext.diff Type: application/octet-stream Size: 2244 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060114/a7e86b5e/attachment.obj From jim at zope.com Sat Jan 14 16:06:11 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 14 Jan 2006 10:06:11 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> Message-ID: <43C91363.7040809@zope.com> Phillip J. Eby wrote: > Hi folks. I just implemented preliminary shared library build support for > setuptools, but I ran into a bit of a snag. I was operating under the > assumption that you could simply put shared libraries in the directory > alongside the extensions that use them, but in practice it turns out that > this only works on Windows. On Linux I found that you could only put > shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar > issue will be found under OS X and other Unixes. > > This seems to be a bit of a problem, in that it appears to mean there's no > sane way to include shared libraries in an egg on any platform other than > Windows. It's common practice for Python extensions built for Windows to > include shared libraries to avoid having to install them in the Windows > system directories, but for Unix-like OSes this isn't really the normal > practice anyway. > > But, in the case where a developer wants to include a shared library as > part of the distribution, this remains somewhat problematic. One can of > course install the library to a platform-appropriate location ($prefix/lib > on most platforms), but to do it from an egg is harder. You can't simply > unzip the egg any more, you would have to also move the libraries to a > standard installation location, breaking any hope of multi-version > support. (And this would have to be able to happen on the fly, when shared > libraries and extensions are extracted to the cache directory, if running > zipped.) > > I don't suppose you can modify LD_LIBRARY_PATH on the fly? Any other > options? Should I just give up on supporting shared libraries on anything > but Windows, perhaps silently converting the shared libraries to static ones? On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the linker at build time that causes the run-time linker to use specific paths to load libraries. This feature is supported by disutils through its --rpath/-R build_ext options. This doesn't work on Mac OS X, but it sounds like someone else may know how to make this work. BTW, I often have need to be able to create what we call "application buildouts" that contain various facilities needed by a particular application. These facilities are mostly Python-based but often include things like relational databases, ICU, ldap servers, etc. We want to be able to have buildouts for multiple applications (or multiple application versions) on a single development machine. Each buildout needs to be able to control versions of things it uses (including Python versions). We try to minimize the dependence on system libraries, as these are usually different on development and deployment systems and because we may need different versions for different applications on a development machine. I'm just beginning to learn about eggs. I certainly hope I'll have the same flexibility when installing eggs that I have now when using distutils. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sat Jan 14 16:11:54 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 14 Jan 2006 10:11:54 -0500 Subject: [Distutils] License for virtual-python.py? Message-ID: <43C914BA.1070704@zope.com> virtual-python.py looks very useful. It lacks a license/copyright statement though. Can we get this? Are there plans to include this script in 2.5? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Jan 14 20:48:37 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 14 Jan 2006 14:48:37 -0500 Subject: [Distutils] 2 platforms down, only Mac to go... In-Reply-To: <50a522ca0601140512h1be4f529g@mail.gmail.com> References: <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060114143643.02246298@mail.telecommunity.com> At 01:12 PM 1/14/2006 +0000, Michael Twomey wrote: >When I ran the tests they all worked, but setuptools had indeed built >a static library. Looking at the compiler output it looks like none of >the LDSHARED or CCSHARED flags you set got used. > >Poking further I found that you are using the dl module to test for >RTLD_NOW, unfortunately dl isn't present on my mac. In fact setup.py >in python's source indicates that it isn't built at all: > >if (dl_inc is not None) and (platform not in ['atheos', 'darwin']): > exts.append( Extension('dl', ['dlmodule.c']) ) > >I temporarily hacked my copy to always set have_rtld to True and I got >errors when setup.py tried to use the flags. Okay, I've changed this so that the 'dl' stuff is only done on non-Mac platforms, and Mac is still recognized as needing stub loaders. > It looks like the >compiler has to be mentioned in the LDSHARED: Fixed in SVN. >Using these flags the module built fine. The otool output doesn't look >quite right though (note how the build directory is mentioned in the >built module, it leads to problems, see below): > >$ otool -L libhellolib.dylib >libhellolib.dylib: > build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib >(compatibility version 0.0.0, current version 0.0.0) > /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current >version 92.0.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, >current version 88.1.2) > >$ otool -L dl-hello.so >dl-hello.so: > build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib >(compatibility version 0.0.0, current version 0.0.0) > /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current >version 92.0.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, >current version 88.1.2) It looks like we probably need to add "-install_name libhellolib.dylib" or maybe "-dylib_install_name libhellolib.dylib" to the library link step for hellolib, in order to get that out of there. >When I comment out the dl stuff in the bootstrap code things look a >lot better, the module imports and runs without problems. >I can move the directory to something_else and it still imports fine. Yeah, it looks like that's only because it used the build directory, relative to the extension's location. :( >If I compile and link by hand all in the single directory then I get a >fully relocatable module free of references to the build directory. >It's looking like the linker is building paths relative to where it's >invoked from. So, you're using the same options supplied by the setup script, just not using paths under build/? Does it still then work with all the test cases? >I'll see if I can tweak the flags a bit, I mightn't be able to come up >with anything but I thought you might like the early feedback. I'm >sure someone with more OS X linking experience than me will come up >with better solutions. I appreciate it! It sounds like we might be getting close. I wonder, though, if perhaps what's happening in the configuration that currently works, is that the path is being interpreted as "bundle relative" to the extension. This is fine and is really what we want, but I'm not sure that it can be done with older Mac OSes. I seem to recall that bundle-relative linking was added in 10.4? Or was it 10.3? I need to start actually *using* a Mac one of these days. :) From pje at telecommunity.com Sat Jan 14 20:56:58 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 14 Jan 2006 14:56:58 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <43C91363.7040809@zope.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060114144852.0233d448@mail.telecommunity.com> At 10:06 AM 1/14/2006 -0500, Jim Fulton wrote: >On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the >linker at build time that causes the run-time linker to use specific >paths to load libraries. This feature is supported by disutils through >its --rpath/-R build_ext options. But that's not relocatable (to different directories), and eggs are intended to be "zero-install" for simple use cases. So, the solution I came up with uses '.' in the rpath, and a stub loader that changes directories long enough to get the extension to load and find its library (or libraries). This appears to work decently enough on Linux. >BTW, I often have need to be able to create what we call "application >buildouts" that contain various facilities needed by a particular >application. These facilities are mostly Python-based but often include >things like relational databases, ICU, ldap servers, etc. We want >to be able to have buildouts for multiple applications (or multiple >application versions) on a single development machine. Each buildout >needs to be able to control versions of things it uses (including >Python versions). We try to minimize the dependence on system libraries, >as these are usually different on development and deployment systems >and because we may need different versions for different applications on >a development machine. I'm just beginning to learn about eggs. >I certainly hope I'll have the same flexibility when installing eggs >that I have now when using distutils. I'm not sure I can answer that question, or that I even know what your question is, actually. :) Eggs don't really try to encompass building arbitrary executables, only Python code, C/C++/Pyrex/SWIG extensions, and libraries linked to the extensions. It sort of sounds like you're saying you'd like to be able to build a bunch of non-Python stuff too, which is likely to remain out of scope for the foreseeable future. If you have managed to shoehorn the distutils into doing it for you, great, but I would be surprised if such extensions would be egg-able in any meaningful way. From jim at zope.com Sat Jan 14 22:01:44 2006 From: jim at zope.com (Jim Fulton) Date: Sat, 14 Jan 2006 16:01:44 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <5.1.1.6.0.20060114144852.0233d448@mail.telecommunity.com> References: <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060114144852.0233d448@mail.telecommunity.com> Message-ID: <43C966B8.5050607@zope.com> Phillip J. Eby wrote: > At 10:06 AM 1/14/2006 -0500, Jim Fulton wrote: > >> On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the >> linker at build time that causes the run-time linker to use specific >> paths to load libraries. This feature is supported by disutils through >> its --rpath/-R build_ext options. > > > But that's not relocatable (to different directories), and eggs are > intended to be "zero-install" for simple use cases. So, the solution I > came up with uses '.' in the rpath, and a stub loader that changes > directories Changes directories? Uh ... this sounds brittle. > long enough to get the extension to load and find its > library (or libraries). This appears to work decently enough on Linux. > > >> BTW, I often have need to be able to create what we call "application >> buildouts" that contain various facilities needed by a particular >> application. These facilities are mostly Python-based but often include >> things like relational databases, ICU, ldap servers, etc. We want >> to be able to have buildouts for multiple applications (or multiple >> application versions) on a single development machine. Each buildout >> needs to be able to control versions of things it uses (including >> Python versions). We try to minimize the dependence on system libraries, >> as these are usually different on development and deployment systems >> and because we may need different versions for different applications on >> a development machine. I'm just beginning to learn about eggs. >> I certainly hope I'll have the same flexibility when installing eggs >> that I have now when using distutils. > > > I'm not sure I can answer that question, or that I even know what your > question is, actually. :) Eggs don't really try to encompass building > arbitrary executables, only Python code, C/C++/Pyrex/SWIG extensions, > and libraries linked to the extensions. It sort of sounds like you're > saying you'd like to be able to build a bunch of non-Python stuff too, > which is likely to remain out of scope for the foreseeable future. No. I want to be able to get a Python egg built in such a way that it uses a library in a location I tell it to, rather than in the standard system libraries. For example, suppose I have an egg with an extension that uses ICU, and that I've installed the ICU libraries in a non-standard place. When I build the egg, I want to tell it to use the location that I specify for ICU. I can do this with setup now with the -R option to build_ext. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sat Jan 14 23:55:25 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 14 Jan 2006 17:55:25 -0500 Subject: [Distutils] Dynamic linking on Linux, Mac OS, et al In-Reply-To: <43C966B8.5050607@zope.com> References: <5.1.1.6.0.20060114144852.0233d448@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060105185020.0223b3a8@mail.telecommunity.com> <5.1.1.6.0.20060114144852.0233d448@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060114175325.023491a8@mail.telecommunity.com> At 04:01 PM 1/14/2006 -0500, Jim Fulton wrote: >Phillip J. Eby wrote: >>At 10:06 AM 1/14/2006 -0500, Jim Fulton wrote: >> >>>On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the >>>linker at build time that causes the run-time linker to use specific >>>paths to load libraries. This feature is supported by disutils through >>>its --rpath/-R build_ext options. >> >>But that's not relocatable (to different directories), and eggs are >>intended to be "zero-install" for simple use cases. So, the solution I >>came up with uses '.' in the rpath, and a stub loader that changes directories > >Changes directories? Uh ... this sounds brittle. Yep. All the solutions are, unfortunately, except on Windows where it just works in a sane way to start with. >No. I want to be able to get a Python egg built in such a way that >it uses a library in a location I tell it to, rather than in the standard >system libraries. For example, suppose I have an egg with an extension that >uses ICU, and that I've installed the ICU libraries in a non-standard place. >When I build the egg, I want to tell it to use the location that I specify >for ICU. I can do this with setup now with the -R option to build_ext. Oh. Sure; you can set it on the command line, e.g. "build_ext -R whatever bdist_egg", or via any of the normal distutils hooks, including the various Extension() keyword arguments, config files, etc. From jim at zope.com Sun Jan 15 16:26:53 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 15 Jan 2006 10:26:53 -0500 Subject: [Distutils] IRC? Message-ID: <43CA69BD.70501@zope.com> Is there an IRC channel that people knowledgable about setuptools hang out on? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sun Jan 15 16:30:06 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 15 Jan 2006 10:30:06 -0500 Subject: [Distutils] setuptools in 2.5? Message-ID: <43CA6A7E.8060306@zope.com> Has a decision been made whether to include setuptools in 2.5? If it is going to be included, then I'll update doctest to use pkg_resources to find doctest text files. If it isn't going to be included then I'll have a quandry. :) I guess that in that case, I'd have to use some conditional logic. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Sun Jan 15 16:46:23 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 15 Jan 2006 10:46:23 -0500 Subject: [Distutils] eggs in Python path Message-ID: <43CA6E4F.3090804@zope.com> The eggs quick guide http://peak.telecommunity.com/DevCenter/PythonEggs#using-eggs says that eggs can be installed by just putting them on sys.path. This doesn't seem to be enough though. A .pth file seem to also be necessary. Am I missing something? Or is the documentation incorrect? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Sun Jan 15 18:23:21 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 15 Jan 2006 12:23:21 -0500 Subject: [Distutils] eggs in Python path In-Reply-To: <43CA6E4F.3090804@zope.com> Message-ID: <5.1.1.6.0.20060115121756.02246920@mail.telecommunity.com> At 10:46 AM 1/15/2006 -0500, Jim Fulton wrote: >The eggs quick guide >http://peak.telecommunity.com/DevCenter/PythonEggs#using-eggs >says that eggs can be installed by just putting them on sys.path. This >doesn't >seem to be enough though. A .pth file seem to also be necessary. > >Am I missing something? Two somethings. :) First, you *can* install an egg by putting it on sys.path. A .pth file is just one *way* of putting it on sys.path. What you're asking about is installing an egg by putting it *in a directory that's on sys.path*, which is a different thing. You can also do this, as long as you're using the pkg_resources API to request the egg, or it's one of your requirements specified by your egg. The only egg that absolutely *must* be in a .pth (or otherwise get onto sys.path) is the setuptools egg. As long as that's the case, then it is not necessary to have .pth files, because the act of require()-ing an egg will cause it and all its dependencies to be added. >Or is the documentation incorrect? It might be a bit confusing about this issue, but I've so far found that changing it around doesn't help much. :( This is simply something so new to most people that they seem to project their existing thought process onto it no matter what it actually says, and then only get it after bumping into a problem or mental contradiction. If you think you can improve upon the comprehensibility without making it so complex that nobody will read it anyway, feel free to submit a patch. :) From pje at telecommunity.com Sun Jan 15 18:34:35 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 15 Jan 2006 12:34:35 -0500 Subject: [Distutils] setuptools in 2.5? In-Reply-To: <43CA6A7E.8060306@zope.com> Message-ID: <5.1.1.6.0.20060115122330.043fdc28@mail.telecommunity.com> At 10:30 AM 1/15/2006 -0500, Jim Fulton wrote: >Has a decision been made whether to include setuptools in 2.5? It hasn't been proposed, and I don't think it's likely. setuptools is still alpha (although I think it's starting to reach beta quality) and there are a lot of features left to implement before 1.0. I do think it's quite possible that quite a lot of setuptools could reasonably be backported to the distutils, including some of its command conveniences (aliases, configuration-saving, distribution rotation, etc.), utility functions (e.g. find_packages), and so on. The parts that are both stable and not specific to eggs, in other words, as these are less likely to be controversial. :) >If it is going to be included, then I'll update doctest to >use pkg_resources to find doctest text files. If it isn't going to >be included then I'll have a quandry. :) I guess that in that case, >I'd have to use some conditional logic. I'd suggest something like this, which will give you a compatible API and will work with any zipped packages, not just eggs: import sys, os def _get_path_and_loader(package_name, resource_name): try: module = sys.modules[moduleName] except KeyError: __import__(moduleName) module = sys.modules[moduleName] loader = getattr(module, '__loader__', None) parts = resource_name.split('/') path = os.path.join(os.path.dirname(module.__file__),*parts) return path, loader def resource_string(package_name, resource_name): path,loader = _get_path_and_loader(package_name, resource_name) if loader is None: f = open(path,'rb') try: return f.read() finally: f.close() else: return loader.get_data(path) try: from pkg_resources import resource_string except ImportError: pass (Note that if 'resource_name' includes a subdirectory path under the associated package, it should use '/' as a separator, not the local os.sep character, as pkg_resources uses distutils-style paths for "resources".) From pje at telecommunity.com Sun Jan 15 18:45:16 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 15 Jan 2006 12:45:16 -0500 Subject: [Distutils] License for virtual-python.py? In-Reply-To: <43C914BA.1070704@zope.com> Message-ID: <5.1.1.6.0.20060114145706.039b2d28@mail.telecommunity.com> At 10:11 AM 1/14/2006 -0500, Jim Fulton wrote: >virtual-python.py looks very useful. It lacks a license/copyright >statement though. Can we get this? Are there plans to include this >script in 2.5? I've got a PSF contributor agreement, so I could check it in. Somebody would need to vouch for maintaining/documenting it, however, as it's only lightly documented at the moment. Mainly, I don't want to be the one to run the Python-Dev gauntlet to get it in. I'm burned out on the "slings and arrows" of the process at this point. If you or somebody else wants to adopt and champion it, great, I salute you. :) From jim at zope.com Sun Jan 15 19:50:44 2006 From: jim at zope.com (Jim Fulton) Date: Sun, 15 Jan 2006 13:50:44 -0500 Subject: [Distutils] eggs in Python path In-Reply-To: <5.1.1.6.0.20060115121756.02246920@mail.telecommunity.com> References: <5.1.1.6.0.20060115121756.02246920@mail.telecommunity.com> Message-ID: <43CA9984.8010202@zope.com> Phillip J. Eby wrote: > At 10:46 AM 1/15/2006 -0500, Jim Fulton wrote: > >> The eggs quick guide >> http://peak.telecommunity.com/DevCenter/PythonEggs#using-eggs >> says that eggs can be installed by just putting them on sys.path. >> This doesn't >> seem to be enough though. A .pth file seem to also be necessary. >> >> Am I missing something? > > > Two somethings. :) > > First, you *can* install an egg by putting it on sys.path. A .pth file > is just one *way* of putting it on sys.path. Ah > What you're asking about is installing an egg by putting it *in a > directory that's on sys.path*, which is a different thing. You can also > do this, as long as you're using the pkg_resources API to request the > egg, or it's one of your requirements specified by your egg. > > The only egg that absolutely *must* be in a .pth (or otherwise get onto > sys.path) is the setuptools egg. As long as that's the case, then it is > not necessary to have .pth files, because the act of require()-ing an > egg will cause it and all its dependencies to be added. Sure, but then you have to call require. :) > >> Or is the documentation incorrect? > > > It might be a bit confusing about this issue, but I've so far found that > changing it around doesn't help much. :( This is simply something so > new to most people that they seem to project their existing thought > process onto it no matter what it actually says, and then only get it > after bumping into a problem or mental contradiction. If you think you > can improve upon the comprehensibility without making it so complex that > nobody will read it anyway, feel free to submit a patch. :) I think that a sentance or 2 saying that when you talk about putting an egg on sys.path or in PYTHONPATH you mean putting the path to the egg file in the list of paths. I suggest that: "If you have a pure-Python egg that doesn't use any in-package data files, and you don't mind manually including the path to the egg file in sys.path or in PYTHONPATH, you can use the egg without installing setuptools. " Would be much clearer. The original text talks about putting the *egg* *on* sys.path or PYTHONPATH. This is missleading in 2 ways: 1. You aren't putting the egg anywhere. You are putting the name of the egg somewhere. 2. When we talk about putting something "on" a path, that suggests (at least to me) that you'll be able to find it in one of the directories specified in the path. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From ianb at colorstudy.com Sun Jan 15 22:16:14 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 15 Jan 2006 15:16:14 -0600 Subject: [Distutils] eggs in Python path In-Reply-To: <5.1.1.6.0.20060115121756.02246920@mail.telecommunity.com> References: <5.1.1.6.0.20060115121756.02246920@mail.telecommunity.com> Message-ID: <43CABB9E.3010905@colorstudy.com> Phillip J. Eby wrote: > At 10:46 AM 1/15/2006 -0500, Jim Fulton wrote: > > >>The eggs quick guide >>http://peak.telecommunity.com/DevCenter/PythonEggs#using-eggs >>says that eggs can be installed by just putting them on sys.path. This >>doesn't >>seem to be enough though. A .pth file seem to also be necessary. >> >>Am I missing something? > > > Two somethings. :) > > First, you *can* install an egg by putting it on sys.path. A .pth file is > just one *way* of putting it on sys.path. I can read that two ways, with Jim's reading being more likely. Maybe it would be better to say that you can install an egg by making sure sys.path points to the egg. Being "on" sys.path can mean either "being in that list of paths" and "contained by a directory on that list of paths." -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From ianb at colorstudy.com Sun Jan 15 22:18:17 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 15 Jan 2006 15:18:17 -0600 Subject: [Distutils] IRC? In-Reply-To: <43CA69BD.70501@zope.com> References: <43CA69BD.70501@zope.com> Message-ID: <43CABC19.1060805@colorstudy.com> Jim Fulton wrote: > Is there an IRC channel that people knowledgable about setuptools hang out on? I hang out on #pythonpaste on freenode, but usually not during work hours. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From rasky at develer.com Mon Jan 16 11:03:58 2006 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 16 Jan 2006 11:03:58 +0100 Subject: [Distutils] setuptools in 2.5? References: <5.1.1.6.0.20060115122330.043fdc28@mail.telecommunity.com> Message-ID: <12ef01c61a84$2b063470$0100a8c0@bagio> Phillip J. Eby wrote: > I do think it's quite possible that quite a lot of setuptools could > reasonably be backported to the distutils, including some of its > command conveniences (aliases, configuration-saving, distribution > rotation, etc.), utility functions (e.g. find_packages), and so on. > The parts that are both stable and not specific to eggs, in other > words, as these are less likely to be controversial. :) That'd be great to do. I think also of fixes to MANIFEST, understanding of cvs/svn metafiles, etc. Are you really planning to have those integrated into distutils for 2.5? Giovanni Bajo From chris at trichech.us Mon Jan 16 16:40:31 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Mon, 16 Jan 2006 10:40:31 -0500 Subject: [Distutils] bundling packages with setuptools? Message-ID: I have a Python module for doing MCMC simulation that requires a number of prerequisite packages to run, all of which are open source (so I can build and distribute the packages that are needed). I am wondering if there is a way to use the power of python eggs to be able to have a single installer that installs or updates all the required packages. Is there a way of having the egg installer for my package go ahead and fetch the eggs of the prerequisite packages and install them? Thanks, C. -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060116/ca8d3842/attachment.bin From dangoor at gmail.com Mon Jan 16 18:20:01 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Mon, 16 Jan 2006 12:20:01 -0500 Subject: [Distutils] bundling packages with setuptools? In-Reply-To: <83137E77-2ACB-4524-AC89-7CC13BC71EC6@trichech.us> References: <3f085ecd0601160854m68379f50t6dc31f2c30fc5926@mail.gmail.com> <83137E77-2ACB-4524-AC89-7CC13BC71EC6@trichech.us> Message-ID: <3f085ecd0601160920v72f5eb16t2e62a60cecc6932e@mail.gmail.com> (readding distutils-sig which I accidentally dropped) On 1/16/06, Christopher Fonnesbeck wrote: > On Jan 16, 2006, at 11:54 AM, Kevin Dangoor wrote: > > > Yes. This is something setuptools does quite admirably. This is in the > > setup.py for TurboGears: > > > > install_requires = ["TurboKid >= 0.9.0", > > "CherryPy > 2.1.1", > > "SQLObject >= 0.7.1dev_r1457", "simplejson >= 1.1", > > "elementtree >= 1.2.6", "PasteScript >= 0.4.1", > > "cElementTree >= 1.0.2", "FormEncode >= 0.4", > > "setuptools >= 0.6a8", > > "RuleDispatch"], > > > > Running easy_install on the TurboGears package will go and fetch all > > of those pre-requisites. > > > > So, easy_install is the installer, and your egg includes the metadata > > about which packages it needs. > > This is what I was hoping. What about URLs? The packages dont have to > be in the cheese shop, do they? Nope. TurboGears uses some packages that aren't in the Cheeseshop. You can use the -f/--find-links option to tell easy_install where to look. You can see the TurboGears installation instructions to see the full process: http://www.turbogears.org/download/index.html Generally, pretty easy. If you have all of the eggs available on your local machine, you can also have setuptools run with those eggs. Kevin From ianb at colorstudy.com Mon Jan 16 18:47:44 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 16 Jan 2006 11:47:44 -0600 Subject: [Distutils] bundling packages with setuptools? In-Reply-To: <3f085ecd0601160920v72f5eb16t2e62a60cecc6932e@mail.gmail.com> References: <3f085ecd0601160854m68379f50t6dc31f2c30fc5926@mail.gmail.com> <83137E77-2ACB-4524-AC89-7CC13BC71EC6@trichech.us> <3f085ecd0601160920v72f5eb16t2e62a60cecc6932e@mail.gmail.com> Message-ID: <43CBDC40.6010001@colorstudy.com> Kevin Dangoor wrote: > (readding distutils-sig which I accidentally dropped) > > On 1/16/06, Christopher Fonnesbeck wrote: > >>On Jan 16, 2006, at 11:54 AM, Kevin Dangoor wrote: >> >> >>>Yes. This is something setuptools does quite admirably. This is in the >>>setup.py for TurboGears: >>> >>> install_requires = ["TurboKid >= 0.9.0", >>> "CherryPy > 2.1.1", >>> "SQLObject >= 0.7.1dev_r1457", "simplejson >= 1.1", >>> "elementtree >= 1.2.6", "PasteScript >= 0.4.1", >>> "cElementTree >= 1.0.2", "FormEncode >= 0.4", >>> "setuptools >= 0.6a8", >>> "RuleDispatch"], >>> >>>Running easy_install on the TurboGears package will go and fetch all >>>of those pre-requisites. >>> >>>So, easy_install is the installer, and your egg includes the metadata >>>about which packages it needs. >> >>This is what I was hoping. What about URLs? The packages dont have to >>be in the cheese shop, do they? > > > Nope. TurboGears uses some packages that aren't in the Cheeseshop. You > can use the -f/--find-links option to tell easy_install where to look. > You can see the TurboGears installation instructions to see the full > process: > > http://www.turbogears.org/download/index.html > > Generally, pretty easy. If you have all of the eggs available on your > local machine, you can also have setuptools run with those eggs. You can also put the links on your Cheese Shop page (in the long_description). If you do that, you should also add the links to setup.cfg, like: [easy_install] find_links = ... The combination of these two I think is sufficient to get the package and dependencies installed both with easy_install and python setup.py install/develop. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From micktwomey at gmail.com Tue Jan 17 00:21:05 2006 From: micktwomey at gmail.com (Michael Twomey) Date: Mon, 16 Jan 2006 23:21:05 +0000 Subject: [Distutils] 2 platforms down, only Mac to go... In-Reply-To: <5.1.1.6.0.20060114143643.02246298@mail.telecommunity.com> References: <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> <50a522ca0601140512h1be4f529g@mail.gmail.com> <5.1.1.6.0.20060114143643.02246298@mail.telecommunity.com> Message-ID: <50a522ca0601161521h367dac12k@mail.gmail.com> Hi again, On 14/01/06, Phillip J. Eby wrote: > > Okay, I've changed this so that the 'dl' stuff is only done on non-Mac > platforms, and Mac is still recognized as needing stub loaders. > Working for me now. > > It looks like the > >compiler has to be mentioned in the LDSHARED: > > Fixed in SVN. > This too. > > It looks like we probably need to add "-install_name libhellolib.dylib" or > maybe "-dylib_install_name libhellolib.dylib" to the library link step for > hellolib, in order to get that out of there. > -install_name did the trick (in my manual compilation anyway). The man page says it all: If this is not specified, the name specified in the -o name option will be used. So this worked: gcc -Wl,-x -dynamiclib -undefined dynamic_lookup $tempdir/hellolib.o \ -install_name libhellolib.dylib -o $libdir/libhellolib.dylib Interestingly you can use "-install_name [any_path]libhellolib.dylib" too, so ../ and ./ combinations are accepted as well. -dylib_install_name didn't work (which surprised me), I got this error: powerpc-apple-darwin8-gcc-4.0.1: libhellolib.dylib: No such file or directory This implies that it was trying to look for libhellolib.dylib before it was built, which is odd. > > >If I compile and link by hand all in the single directory then I get a > >fully relocatable module free of references to the build directory. > >It's looking like the linker is building paths relative to where it's > >invoked from. > > So, you're using the same options supplied by the setup script, just not > using paths under build/? Does it still then work with all the test cases? > I copied the output of distutils into a shell script and modified by hand. Based on the comment in the man page above it was using the name from the -o flag which is why it worked when I built manually. It worked in the test cases. > > >I'll see if I can tweak the flags a bit, I mightn't be able to come up > >with anything but I thought you might like the early feedback. I'm > >sure someone with more OS X linking experience than me will come up > >with better solutions. > > I appreciate it! It sounds like we might be getting close. > It looks like with the libhellolib.dylib built with the "-install_name libhellolib.dylib" does the trick. Everything works fine when I try the various tests. I've attached the shell script I was using to build and test for reference. > I wonder, though, if perhaps what's happening in the configuration that > currently works, is that the path is being interpreted as "bundle relative" > to the extension. This is fine and is really what we want, but I'm not > sure that it can be done with older Mac OSes. I seem to recall that > bundle-relative linking was added in 10.4? Or was it 10.3? I need to > start actually *using* a Mac one of these days. :) > I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), I'm pretty sure it was the "-undefined dynamic_lookup" which got introduced, which had a big impact on shared libraries in bundles. I'm not sure how this would affect the linking stuff above, it is "used to allow any undefined symbols to be looked up dynamically at runtime" according to the man page. cheers, Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: go.sh Type: application/x-sh Size: 1209 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060116/8ac4dc78/attachment.sh From bob at redivi.com Tue Jan 17 00:32:26 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 16 Jan 2006 15:32:26 -0800 Subject: [Distutils] 2 platforms down, only Mac to go... In-Reply-To: <50a522ca0601161521h367dac12k@mail.gmail.com> References: <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> <50a522ca0601140512h1be4f529g@mail.gmail.com> <5.1.1.6.0.20060114143643.02246298@mail.telecommunity.com> <50a522ca0601161521h367dac12k@mail.gmail.com> Message-ID: On Jan 16, 2006, at 3:21 PM, Michael Twomey wrote: > On 14/01/06, Phillip J. Eby wrote: >> >> I wonder, though, if perhaps what's happening in the configuration >> that >> currently works, is that the path is being interpreted as "bundle >> relative" >> to the extension. This is fine and is really what we want, but >> I'm not >> sure that it can be done with older Mac OSes. I seem to recall that >> bundle-relative linking was added in 10.4? Or was it 10.3? I >> need to >> start actually *using* a Mac one of these days. :) >> > > I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), > I'm pretty sure it was the "-undefined dynamic_lookup" which got > introduced, which had a big impact on shared libraries in bundles. I'm > not sure how this would affect the linking stuff above, it is "used to > allow any undefined symbols to be looked up dynamically at runtime" > according to the man page. You are correct, MACOSX_DEPLOYMENT_TARGET=10.3 allows you to use - undefined dynamic_lookup. However, requiring Mac OS X 10.4 would allow you to use @loader_path [1], which would let you skip out on the chdir hack. [1] http://developer.apple.com/releasenotes/DeveloperTools/dyld.html -bob From pje at telecommunity.com Tue Jan 17 02:59:33 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 16 Jan 2006 20:59:33 -0500 Subject: [Distutils] bundling packages with setuptools? In-Reply-To: Message-ID: <5.1.1.6.0.20060116205511.00aae070@mail.telecommunity.com> At 10:40 AM 01/16/2006 -0500, Christopher Fonnesbeck wrote: >I have a Python module for doing MCMC simulation that requires a >number of prerequisite packages to run, all of which are open source >(so I can build and distribute the packages that are needed). I am >wondering if there is a way to use the power of python eggs to be >able to have a single installer that installs or updates all the >required packages. Is there a way of having the egg installer for my >package go ahead and fetch the eggs of the prerequisite packages and >install them? I see some other folks have answered this in part; for more information, you can also see the setuptools manual at http://peak.telecommunity.com/DevCenter/setuptools, which describes in quite a bit of detail how to specify dependencies, what your users should know, etc. From pje at telecommunity.com Tue Jan 17 03:09:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 16 Jan 2006 21:09:20 -0500 Subject: [Distutils] 2 platforms down, only Mac to go... In-Reply-To: References: <50a522ca0601161521h367dac12k@mail.gmail.com> <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> <50a522ca0601140512h1be4f529g@mail.gmail.com> <5.1.1.6.0.20060114143643.02246298@mail.telecommunity.com> <50a522ca0601161521h367dac12k@mail.gmail.com> Message-ID: <5.1.1.6.0.20060116210433.00a872b0@mail.telecommunity.com> At 03:32 PM 01/16/2006 -0800, Bob Ippolito wrote: >On Jan 16, 2006, at 3:21 PM, Michael Twomey wrote: > >>On 14/01/06, Phillip J. Eby wrote: >>> >>>I wonder, though, if perhaps what's happening in the configuration >>>that >>>currently works, is that the path is being interpreted as "bundle >>>relative" >>>to the extension. This is fine and is really what we want, but >>>I'm not >>>sure that it can be done with older Mac OSes. I seem to recall that >>>bundle-relative linking was added in 10.4? Or was it 10.3? I >>>need to >>>start actually *using* a Mac one of these days. :) >> >>I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), >>I'm pretty sure it was the "-undefined dynamic_lookup" which got >>introduced, which had a big impact on shared libraries in bundles. I'm >>not sure how this would affect the linking stuff above, it is "used to >>allow any undefined symbols to be looked up dynamically at runtime" >>according to the man page. > >You are correct, MACOSX_DEPLOYMENT_TARGET=10.3 allows you to use - >undefined dynamic_lookup. However, requiring Mac OS X 10.4 would >allow you to use @loader_path [1], which would let you skip out on >the chdir hack. > >[1] http://developer.apple.com/releasenotes/DeveloperTools/dyld.html So, should setuptools maybe check the deployment target and use that to decide whether to chdir-hack or to use -install_name @loader_path/blah.dylib? That won't make the current platform tests and build customization any more complex on a fundamental level. I already have to decide whether to use a chdir loader (not needed at all on Windows) and whether the loader should use dl (Linux) or not (Mac). So, adding an environment variable check doesn't sound like a big deal. Do any OS X versions prior to 10.3 even have Python installed, btw? I'm figuring I can live with not supporting those, though I should double-check what OSAF is targeting, of course. From bob at redivi.com Tue Jan 17 02:45:53 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 16 Jan 2006 17:45:53 -0800 Subject: [Distutils] 2 platforms down, only Mac to go... In-Reply-To: <5.1.1.6.0.20060116210433.00a872b0@mail.telecommunity.com> References: <50a522ca0601161521h367dac12k@mail.gmail.com> <5.1.1.6.0.20060113194115.01e472e0@mail.telecommunity.com> <50a522ca0601140512h1be4f529g@mail.gmail.com> <5.1.1.6.0.20060114143643.02246298@mail.telecommunity.com> <50a522ca0601161521h367dac12k@mail.gmail.com> <5.1.1.6.0.20060116210433.00a872b0@mail.telecommunity.com> Message-ID: <65EFB12C-05FA-4C9B-BD80-4C1DEB31AD84@redivi.com> On Jan 16, 2006, at 6:09 PM, Phillip J. Eby wrote: > At 03:32 PM 01/16/2006 -0800, Bob Ippolito wrote: > >> On Jan 16, 2006, at 3:21 PM, Michael Twomey wrote: >> >>> On 14/01/06, Phillip J. Eby wrote: >>>> >>>> I wonder, though, if perhaps what's happening in the configuration >>>> that >>>> currently works, is that the path is being interpreted as "bundle >>>> relative" >>>> to the extension. This is fine and is really what we want, but >>>> I'm not >>>> sure that it can be done with older Mac OSes. I seem to recall >>>> that >>>> bundle-relative linking was added in 10.4? Or was it 10.3? I >>>> need to >>>> start actually *using* a Mac one of these days. :) >>> >>> I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), >>> I'm pretty sure it was the "-undefined dynamic_lookup" which got >>> introduced, which had a big impact on shared libraries in >>> bundles. I'm >>> not sure how this would affect the linking stuff above, it is >>> "used to >>> allow any undefined symbols to be looked up dynamically at runtime" >>> according to the man page. >> >> You are correct, MACOSX_DEPLOYMENT_TARGET=10.3 allows you to use - >> undefined dynamic_lookup. However, requiring Mac OS X 10.4 would >> allow you to use @loader_path [1], which would let you skip out on >> the chdir hack. >> >> [1] http://developer.apple.com/releasenotes/DeveloperTools/dyld.html > > So, should setuptools maybe check the deployment target and use > that to decide whether to chdir-hack or to use -install_name > @loader_path/blah.dylib? That won't make the current platform > tests and build customization any more complex on a fundamental > level. I already have to decide whether to use a chdir loader (not > needed at all on Windows) and whether the loader should use dl > (Linux) or not (Mac). So, adding an environment variable check > doesn't sound like a big deal. Apple has actually shifted their recommended runtime dynamic linkage API to plain old dl.. it's part of libc in Mac OS X 10.4, available for dynamic linkage in Mac OS X 10.3, and provided as a static library for earlier versions of OS X. It might be worth considering shifting Python 2.5 towards that API instead of the old NeXT-style APIs. I don't want to do that work, of course, the APIs are equivalent and what we have works. Personally I'd probably just go with the chdir loader for now, because people are still running 10.3. In a year or two, it might be worth dropping 10.3 support and switching over to @loader_path and removing the cruft. Until then, I don't see the benefit of maintaining two code paths. > Do any OS X versions prior to 10.3 even have Python installed, > btw? I'm figuring I can live with not supporting those, though I > should double-check what OSAF is targeting, of course. OSAF is not going to want to target any vendor-installed Python. They should be bundling a self-contained Python installation with Chandler (as I'm sure they do on Windows, too). If they do depend on a vendor-installed Python, they'll have to have separate versions of everything for almost every OS release. I wouldn't even worry about it. To answer your question, 10.2 had a horrifically bad build of Python 2.2.0. -bob From pje at telecommunity.com Tue Jan 17 20:39:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 17 Jan 2006 14:39:57 -0500 Subject: [Distutils] Sourceforge changed HTML again... Message-ID: <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> Hi all. Ian Bicking reported an issue with Sourceforge's download process changing again; I've updated setuptools in SVN but haven't made a new release yet. If you need the fixed version, update to the development version via "ez_setup.py setuptools==dev" for now. Thanks. From rasky at develer.com Tue Jan 17 19:54:32 2006 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 17 Jan 2006 19:54:32 +0100 Subject: [Distutils] Sourceforge changed HTML again... References: <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> Message-ID: <02e401c61b97$73ec3770$bf03030a@trilan> Phillip J. Eby wrote: > Hi all. Ian Bicking reported an issue with Sourceforge's download process > changing again; I've updated setuptools in SVN but haven't made a new > release yet. If you need the fixed version, update to the development > version via "ez_setup.py setuptools==dev" for now. Thanks. This is a probably a FAQ, but maybe the Sourceforge guys can be talked into providing some machine-friendly way for downloading files? -- Giovanni Bajo From dangoor at gmail.com Tue Jan 17 20:07:12 2006 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 17 Jan 2006 14:07:12 -0500 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <02e401c61b97$73ec3770$bf03030a@trilan> References: <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> Message-ID: <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> On 1/17/06, Giovanni Bajo wrote: > Phillip J. Eby wrote: > > > > Hi all. Ian Bicking reported an issue with Sourceforge's download process > > changing again; I've updated setuptools in SVN but haven't made a new > > release yet. If you need the fixed version, update to the development > > version via "ez_setup.py setuptools==dev" for now. Thanks. > > This is a probably a FAQ, but maybe the Sourceforge guys can be talked into > providing some machine-friendly way for downloading files? The problem is that a machine doesn't click on ads. (Or, at least, isn't supposed to :) Kevin From ianb at colorstudy.com Tue Jan 17 21:09:55 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 17 Jan 2006 14:09:55 -0600 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> References: <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> Message-ID: <43CD4F13.90007@colorstudy.com> Kevin Dangoor wrote: >>This is a probably a FAQ, but maybe the Sourceforge guys can be talked into >>providing some machine-friendly way for downloading files? > > > The problem is that a machine doesn't click on ads. (Or, at least, > isn't supposed to :) Since they have a 5-second delay in downloading, I assume they really are trying to get ad clicks (since 1 or 0 seconds would do just as well, and direct links would of course be better for everyone). OTOH, it never hurts to ask; maybe someone can open a ticket over on SF? If anyone does, please note that here. Hell, a stable API with a request that API users show a text ad (that would be delivered as part of the API) would be fine with me (as a user of easy_install). But right now I think it is more useful to encourage people to use the cheese shop for file distribution -- SF is also really hard to upload to, so the cheese shop is better on both ends. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Jan 18 01:53:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 17 Jan 2006 19:53:41 -0500 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <02e401c61b97$73ec3770$bf03030a@trilan> References: <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060117194917.00a7c7b0@mail.telecommunity.com> At 07:54 PM 01/17/2006 +0100, Giovanni Bajo wrote: >Phillip J. Eby wrote: > > Hi all. Ian Bicking reported an issue with Sourceforge's download process > > changing again; I've updated setuptools in SVN but haven't made a new > > release yet. If you need the fixed version, update to the development > > version via "ez_setup.py setuptools==dev" for now. Thanks. > >This is a probably a FAQ, but maybe the Sourceforge guys can be talked into >providing some machine-friendly way for downloading files? They have already done it, and I've changed setuptools to use it now (as of r42088). I've now removed all of the code originally contributed by Ian to parse SourceForge's "human user" HTML and simply preprocess prdownloads.sf.net URLs to a similar path on dl.sourceforge.net, which is a round-robin DNS for the various mirrors. This cuts two page fetches out of the process and eliminates any HTML scraping dependencies. As long as SF doesn't get rid of the *.dl system, this should continue to work no matter how much the HTML frontend changes. From pje at telecommunity.com Wed Jan 18 02:04:14 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 17 Jan 2006 20:04:14 -0500 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <43CD4F13.90007@colorstudy.com> References: <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> Message-ID: <5.1.1.6.0.20060117195446.024c0bb0@mail.telecommunity.com> At 02:09 PM 01/17/2006 -0600, Ian Bicking wrote: >Since they have a 5-second delay in downloading, I assume they really >are trying to get ad clicks (since 1 or 0 seconds would do just as well, >and direct links would of course be better for everyone). OTOH, it >never hurts to ask; maybe someone can open a ticket over on SF? If >anyone does, please note that here. Hell, a stable API with a request >that API users show a text ad (that would be delivered as part of the >API) would be fine with me (as a user of easy_install). I checked what the NetBSD "pkgsrc" system does, and it uses the fact that there is a .dl.sourceforge.net subdomain for mirrors. I investigated further and found that dl.sourceforge.net is a round-robin (or random?) DNS for each of the mirrors in the subdomain, so there's a simple transformation from the user-visible download pages to the actual download address. You can see this in the new fix_sf_url() function I added to setuptools.package_index. >But right now I think it is more useful to encourage people to use the >cheese shop for file distribution -- SF is also really hard to upload >to, so the cheese shop is better on both ends. Yeah, uploading to cheeseshop is great for new projects. But practically speaking, SF *is* used to distribute a lot of things, especially big things that mirrors are more useful for. So, it's definitely good for stuff to work. :) From cwmoad at gmail.com Wed Jan 18 19:35:41 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Wed, 18 Jan 2006 13:35:41 -0500 Subject: [Distutils] namespace_packages Message-ID: <6382066a0601181035u46e1345es8272877ca5245169@mail.gmail.com> Following the instruction for setuptools, I am trying to make matplotlib and basemap (a mpl toolkit) share the namespace "matplotlib/toolkits". I can build and install the eggs no problems. "matplotlib/toolkits" is in both's "EGG-INFO/namespace_packages.txt" files. I have added __init__.py files with "__import__('pkg_resources').declare_namespace(__name__)" in both the mpl and basemap matplotlib/toolkits folders. Basically I have done a ton of playing around and "from matplotlib.toolkits import basemap" will not work. It is always limited to the scope of the matplotlib egg which it is hitting first. Are the docs not reflective of the 0.6a9 release? Any help is greatly appreciated. Thanks, Charlie From robert.kern at gmail.com Wed Jan 18 19:51:34 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Jan 2006 12:51:34 -0600 Subject: [Distutils] namespace_packages In-Reply-To: <6382066a0601181035u46e1345es8272877ca5245169@mail.gmail.com> References: <6382066a0601181035u46e1345es8272877ca5245169@mail.gmail.com> Message-ID: Charlie Moad wrote: > Following the instruction for setuptools, I am trying to make > matplotlib and basemap (a mpl toolkit) share the namespace > "matplotlib/toolkits". I can build and install the eggs no problems. > "matplotlib/toolkits" is in both's "EGG-INFO/namespace_packages.txt" > files. I have added __init__.py files with > "__import__('pkg_resources').declare_namespace(__name__)" in both the > mpl and basemap matplotlib/toolkits folders. Basically I have done a > ton of playing around and "from matplotlib.toolkits import basemap" > will not work. It is always limited to the scope of the matplotlib > egg which it is hitting first. Are the docs not reflective of the > 0.6a9 release? Any help is greatly appreciated. That approach (adding the declare_namespace() calls to __init__.py files) should only be necessary when the __init__.py files contain actual code that needs to be executed. In this case, matplotlib/toolkits/__init__.py are otherwise empty, IIRC. You should just add "matplotlib.toolkits" to the namespace_packages.txt. Note the spelling of that. "." not "/". -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pje at telecommunity.com Wed Jan 18 21:08:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 18 Jan 2006 15:08:36 -0500 Subject: [Distutils] namespace_packages In-Reply-To: References: <6382066a0601181035u46e1345es8272877ca5245169@mail.gmail.com> <6382066a0601181035u46e1345es8272877ca5245169@mail.gmail.com> Message-ID: <5.1.1.6.0.20060118150542.00ac83d0@mail.telecommunity.com> At 12:51 PM 01/18/2006 -0600, Robert Kern wrote: >Charlie Moad wrote: > > Following the instruction for setuptools, I am trying to make > > matplotlib and basemap (a mpl toolkit) share the namespace > > "matplotlib/toolkits". I can build and install the eggs no problems. > > "matplotlib/toolkits" is in both's "EGG-INFO/namespace_packages.txt" > > files. I have added __init__.py files with > > "__import__('pkg_resources').declare_namespace(__name__)" in both the > > mpl and basemap matplotlib/toolkits folders. Basically I have done a > > ton of playing around and "from matplotlib.toolkits import basemap" > > will not work. It is always limited to the scope of the matplotlib > > egg which it is hitting first. Are the docs not reflective of the > > 0.6a9 release? Any help is greatly appreciated. > >That approach (adding the declare_namespace() calls to __init__.py files) >should >only be necessary when the __init__.py files contain actual code that needs to >be executed. Namespace packages will *require* declare_namespace() in __init__.py as of 0.7. Currently, there is a warning if you do not include it, because it will absolutely be required in 0.7. The problem with the earlier approach of leaving it out is that namespace packages were only being activated when pkg_resources was imported, and *all* namespace packages on sys.path were being immediately activated. These things are both somewhat problematic, so in 0.7 the declare_namespace() approach will fix them. For the remainder of 0.6, the old behavior remains, but there is a warning so that everybody will add a declare_namespace() by the time 0.7 releases start coming out. From pje at telecommunity.com Wed Jan 18 21:05:26 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 18 Jan 2006 15:05:26 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <6382066a0601181035u46e1345es8272877ca5245169@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> At 01:35 PM 01/18/2006 -0500, Charlie Moad wrote: >Following the instruction for setuptools, I am trying to make >matplotlib and basemap (a mpl toolkit) share the namespace >"matplotlib/toolkits". I can build and install the eggs no problems. >"matplotlib/toolkits" is in both's "EGG-INFO/namespace_packages.txt" It should be matplotlib.toolkits, not matplotlib/toolkits. >files. I have added __init__.py files with >"__import__('pkg_resources').declare_namespace(__name__)" in both the >mpl and basemap matplotlib/toolkits folders. Basically I have done a >ton of playing around and "from matplotlib.toolkits import basemap" >will not work. It is always limited to the scope of the matplotlib >egg which it is hitting first. Did you run "setup.py develop" or "setup.py install" in both projects? > Are the docs not reflective of the >0.6a9 release? Any help is greatly appreciated. AFAIK, the docs are correct. If you can point me to the source code for your packages I can take a look at them. From ianb at colorstudy.com Thu Jan 19 05:23:45 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 18 Jan 2006 22:23:45 -0600 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <5.1.1.6.0.20060117195446.024c0bb0@mail.telecommunity.com> References: <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> <5.1.1.6.0.20060117195446.024c0bb0@mail.telecommunity.com> Message-ID: <43CF1451.1030600@colorstudy.com> Phillip J. Eby wrote: > I checked what the NetBSD "pkgsrc" system does, and it uses the fact > that there is a .dl.sourceforge.net subdomain for mirrors. I > investigated further and found that dl.sourceforge.net is a round-robin > (or random?) DNS for each of the mirrors in the subdomain, so there's a > simple transformation from the user-visible download pages to the actual > download address. You can see this in the new fix_sf_url() function I > added to setuptools.package_index. I just got a Connection Refused error, but it worked on the second try. As I remember, the port system typically has a set of links, and frequently fails over from one link to the next, so I expect that this error should be expected. When it is encountered, setuptools should just try again until it works. I have noticed in the past that SF mirrors go up and down quite frequently, probably too fast for DNS to keep up. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From frederic.gobry at epfl.ch Thu Jan 19 14:11:58 2006 From: frederic.gobry at epfl.ch (=?iso-8859-1?Q?Fr=E9d=E9ric?= Gobry) Date: Thu, 19 Jan 2006 14:11:58 +0100 Subject: [Distutils] making eggs and debs live together peacefully Message-ID: <20060119131158.GB7025@mnementh.homelinux.org> Hi, I've started using setuptools as a developer quite recently, and appreciate the extra features. I've a question though, regarding the way dependencies are resolved when installing such a package: I've written a package that uses elementtree, and would have found natural to add it in install_requires. However, I've already installed it as a debian package, and in the current state setuptools tries to reinstall over it. So, should I add some extra logic to the setup.py script (ie, try the dependencies myself) or is there a way to tell it "don't update the dependence if you can already import the module" in a convenient way? Fr?d?ric -- Fr?d?ric Gobry Infoscience SISB / EPFL Tel: +41216932288 http://people.epfl.ch/frederic.gobry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060119/bb21b797/attachment.pgp From jeff.pitman at gmail.com Thu Jan 19 15:03:10 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Thu, 19 Jan 2006 06:03:10 -0800 Subject: [Distutils] making eggs and debs live together peacefully In-Reply-To: <20060119131158.GB7025@mnementh.homelinux.org> References: <20060119131158.GB7025@mnementh.homelinux.org> Message-ID: <6b9c17630601190603n48874466q125de98c521f5f4d@mail.gmail.com> On 1/19/06, Fr?d?ric Gobry wrote: > So, should I add some extra logic to the setup.py script (ie, try the > dependencies myself) or is there a way to tell it "don't update the > dependence if you can already import the module" in a convenient way? The best way is going to convince deb packagers to include the egg info so that setuptools can see that the package is in place. Each package comes with its own meta-data. I'm certain Fedora will be heading in this direction. In the interim, it'll be kind of painful. -- -jeff From frederic.gobry at epfl.ch Thu Jan 19 15:48:18 2006 From: frederic.gobry at epfl.ch (=?iso-8859-1?Q?Fr=E9d=E9ric?= Gobry) Date: Thu, 19 Jan 2006 15:48:18 +0100 Subject: [Distutils] making eggs and debs live together peacefully In-Reply-To: <6b9c17630601190603n48874466q125de98c521f5f4d@mail.gmail.com> References: <20060119131158.GB7025@mnementh.homelinux.org> <6b9c17630601190603n48874466q125de98c521f5f4d@mail.gmail.com> Message-ID: <20060119144817.GC7025@mnementh.homelinux.org> > The best way is going to convince deb packagers to include the egg > info so that setuptools can see that the package is in place. Each > package comes with its own meta-data. Indeed, this looks like the right way (tm), but in the meantime wouldn't it be reasonable to include a transition mechanism? > I'm certain Fedora will be heading in this direction. I'll try to see if there is something pending for debian (python-setuptools is part of testing, so...) [...dig dig dig...] Yuck, I hope to find something more positive than that: :-( Fr?d?ric -- Fr?d?ric Gobry Infoscience SISB / EPFL Tel: +41216932288 http://people.epfl.ch/frederic.gobry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060119/21805a07/attachment.pgp From jeff.pitman at gmail.com Thu Jan 19 16:17:42 2006 From: jeff.pitman at gmail.com (Jeff Pitman) Date: Thu, 19 Jan 2006 07:17:42 -0800 Subject: [Distutils] making eggs and debs live together peacefully In-Reply-To: <20060119144817.GC7025@mnementh.homelinux.org> References: <20060119131158.GB7025@mnementh.homelinux.org> <6b9c17630601190603n48874466q125de98c521f5f4d@mail.gmail.com> <20060119144817.GC7025@mnementh.homelinux.org> Message-ID: <6b9c17630601190717q1971e1d8q3a392ba6bd500da8@mail.gmail.com> On 1/19/06, Fr?d?ric Gobry wrote: > > I'm certain Fedora will be heading in this direction. > > I'll try to see if there is something pending for debian > (python-setuptools is part of testing, so...) > > [...dig dig dig...] > > Yuck, I hope to find something more positive than that: > > > > :-( I didn't read the whole thing, but, I'm presuming that it doesn't preclude you from providing a patch that demos what you're looking for. It's just that the authors are working towards the ideal rather than putting patchwork in for every known packaging situation. And, since setuptools may not even be in 2.5 and it's in a alpha state, integration appears to be a non-goal. Other systems conforming *to* setuptools will be exponentially easier than setuptools conforming to others. Of course, the turn-around for debian changes can cause an equilibrium to this difficulty equation... -- -jeff From cwmoad at gmail.com Thu Jan 19 17:44:17 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Thu, 19 Jan 2006 11:44:17 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> Message-ID: <6382066a0601190844n4644acf4p6a18ed445ae8eb82@mail.gmail.com> I tried the "." and I keep getting: error in basemap setup command: Distribution contains no modules or packages for namespace package 'matplotlib.toolkits' On 1/18/06, Phillip J. Eby wrote: > At 01:35 PM 01/18/2006 -0500, Charlie Moad wrote: > >Following the instruction for setuptools, I am trying to make > >matplotlib and basemap (a mpl toolkit) share the namespace > >"matplotlib/toolkits". I can build and install the eggs no problems. > >"matplotlib/toolkits" is in both's "EGG-INFO/namespace_packages.txt" > > It should be matplotlib.toolkits, not matplotlib/toolkits. > > > >files. I have added __init__.py files with > >"__import__('pkg_resources').declare_namespace(__name__)" in both the > >mpl and basemap matplotlib/toolkits folders. Basically I have done a > >ton of playing around and "from matplotlib.toolkits import basemap" > >will not work. It is always limited to the scope of the matplotlib > >egg which it is hitting first. > > Did you run "setup.py develop" or "setup.py install" in both projects? > > > > Are the docs not reflective of the > >0.6a9 release? Any help is greatly appreciated. > > AFAIK, the docs are correct. If you can point me to the source code for > your packages I can take a look at them. > > From micktwomey at gmail.com Thu Jan 19 17:59:51 2006 From: micktwomey at gmail.com (Michael Twomey) Date: Thu, 19 Jan 2006 16:59:51 +0000 Subject: [Distutils] OS X problem with pkg_resources and the twisted event loop Message-ID: <50a522ca0601190859y42f216c8y@mail.gmail.com> Hi, I've come across a problem with an easy_install managed package (matplotlib) on my OS X box. When I try to import it for the first time inside of a twisted application I get this nasty traceback: Traceback (most recent call last): File "/Users/mtt/Desktop/twisted_macosx_vers.py", line 12, in ? reactor.run() File "/usr/local/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 206, in run self.mainLoop() File "/usr/local/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 214, in mainLoop self.runUntilCurrent() --- --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/base.py", line 541, in runUntilCurrent call.func(*call.args, **call.kw) File "/Users/mtt/Desktop/twisted_macosx_vers.py", line 4, in get_platform import pkg_resources File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/pkg_resources.py", line 534, in ? class Environment(object): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/pkg_resources.py", line 537, in Environment def __init__(self,search_path=None,platform=get_platform(),python=PY_MAJOR): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/pkg_resources.py", line 147, in get_platform version = _macosx_vers() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/pkg_resources.py", line 126, in _macosx_vers info = os.popen('/usr/bin/sw_vers').read().splitlines() exceptions.IOError: [Errno 4] Interrupted system call This seems to be caused by twisted's signal handling, which breaks os.popen (and subprocess), see this bug: http://twistedmatrix.com/bugs/issue733 I've attached a simple test script which triggers the problem (most of the time for me, you have to run it a few times as it doesn't always crash). I've got a work around though: If I import pkg_resources before starting my twisted app it seems to fix the problem. (I presume that get_platform's output is being kept somewhere, Environment's __init__ args I think.) cheers, Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: twisted_macosx_vers.py Type: application/octet-stream Size: 237 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060119/8deb617a/attachment.obj From ianb at colorstudy.com Thu Jan 19 18:04:54 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 19 Jan 2006 11:04:54 -0600 Subject: [Distutils] making eggs and debs live together peacefully In-Reply-To: <20060119131158.GB7025@mnementh.homelinux.org> References: <20060119131158.GB7025@mnementh.homelinux.org> Message-ID: <43CFC6B6.3020904@colorstudy.com> Fr?d?ric Gobry wrote: > I've started using setuptools as a developer quite recently, and > appreciate the extra features. I've a question though, regarding the way > dependencies are resolved when installing such a package: I've written a > package that uses elementtree, and would have found natural to add it in > install_requires. However, I've already installed it as a debian > package, and in the current state setuptools tries to reinstall over it. Note that it doesn't install *over* it at all. A normal "python setup.py install" without setuptools and with Debian's default distutils.cfg *will* install over it, but with setuptools/easy_install it will not. If you install multiversion (-m to easy_install) then you won't really have effected the system at all except for packages that specifically require ElementTree through setuptools mechanisms. Altogether it is much lower impact than normal distutils installs. > So, should I add some extra logic to the setup.py script (ie, try the > dependencies myself) or is there a way to tell it "don't update the > dependence if you can already import the module" in a convenient way? There is a new option, added largely for Debian and other packagers, of python setup.py install --single-version-externally-managed. However, this doesn't add the information to an already-installed package, instead it allows package maintainers to add that egg information without otherwise effecting the layout they are using for libraries. It would almost be reasonable to add this metadata to a package that is already installed, except that I don't know how setuptools could figure out the metadata (version, distribution name) except through Debian-specific hooks. Though it is not unreasonable to write the files by hand -- I think all you need is a directory like ElementTree-VERSION.egg-info/, with one or two files in it. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Jan 19 18:45:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 19 Jan 2006 12:45:57 -0500 Subject: [Distutils] making eggs and debs live together peacefully In-Reply-To: <20060119131158.GB7025@mnementh.homelinux.org> Message-ID: <5.1.1.6.0.20060119123806.00a4b6a0@mail.telecommunity.com> At 02:11 PM 01/19/2006 +0100, Fr?d?ric Gobry wrote: >Hi, > >I've started using setuptools as a developer quite recently, and >appreciate the extra features. I've a question though, regarding the way >dependencies are resolved when installing such a package: I've written a >package that uses elementtree, and would have found natural to add it in >install_requires. However, I've already installed it as a debian >package, and in the current state setuptools tries to reinstall over it. On Debian-based systems, you should add these lines to ~/.pydistutils.cfg if you want to install eggs system-wide (substituting python2.4 if appropriate): [install] prefix=/usr/local [easy_install] site_dirs=/usr/local/lib/python2.3/site-packages This will ensure that easy_install does *not* try to overwrite system packages, although it may duplicate them in the interim. For more information, please see the "Custom Installation Locations" section of the EasyInstall wiki page, which describes other ways of customizing where and how packages are installed. This will of course duplicate system packages that do not carry egg metadata. If you want to make it possible to use the Debian-installed packages without duplication, you can do this by installing .egg-info files with the same base filename as the egg they would represent, placed in the site-packages directory where the package is installed. So, you might have something like "elementtree-1.0.2-py2.3.egg-info" in site-packages. The file's contents should be the package's PKG-INFO as created by the distutils. The setuptools runtime will find these files and thus know that it can access those packages as long as 'site-packages' is listed in sys.path. From pje at telecommunity.com Thu Jan 19 18:49:49 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 19 Jan 2006 12:49:49 -0500 Subject: [Distutils] OS X problem with pkg_resources and the twisted event loop In-Reply-To: <50a522ca0601190859y42f216c8y@mail.gmail.com> Message-ID: <5.1.1.6.0.20060119124827.025389d0@mail.telecommunity.com> At 04:59 PM 01/19/2006 +0000, Michael Twomey wrote: > I've got a work around though: If I import pkg_resources before starting > my twisted app it seems to fix the problem. (I presume that > get_platform's output is being kept somewhere, Environment's __init__ > args I think.) cheers, Michae Yes, in general it's best to import pkg_resources sooner rather than later. :) If you're running an application using a script generated by easy_install, this is taken care of for you, as pkg_resources gets imported in order to set up sys.path with the script's requirements. From pje at telecommunity.com Thu Jan 19 18:53:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 19 Jan 2006 12:53:12 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <6382066a0601190844n4644acf4p6a18ed445ae8eb82@mail.gmail.co m> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> At 11:44 AM 01/19/2006 -0500, Charlie Moad wrote: >I tried the "." and I keep getting: > >error in basemap setup command: Distribution contains no modules or >packages for namespace package 'matplotlib.toolkits' Great, now we're making progress. :) Can you post your setup.py? It sounds like you don't have 'matplotlib.toolkits' listed in your 'packages' keyword, or perhaps there's something amiss with your package_dir or something. From tanner at real-time.com Thu Jan 19 18:36:56 2006 From: tanner at real-time.com (Bob Tanner) Date: Thu, 19 Jan 2006 11:36:56 -0600 Subject: [Distutils] making eggs and debs live together peacefully In-Reply-To: <20060119144817.GC7025@mnementh.homelinux.org> References: <20060119131158.GB7025@mnementh.homelinux.org> <6b9c17630601190603n48874466q125de98c521f5f4d@mail.gmail.com> <20060119144817.GC7025@mnementh.homelinux.org> Message-ID: <200601191136.56695.tanner@real-time.com> On Thursday 19 January 2006 08:48, Fr?d?ric Gobry wrote: > Yuck, I hope to find something more positive than that: > > ? > That post summarizes a very long thread of rants between debian people and distutils. I asked the same question you are asking. Lots of ranting on the debian-python mailing list and ultimately no recommendations or answers. The rant thread is here: http://lists.debian.org/debian-python/2005/11/msg00008.html It looks like something positive is starting to happen, you can read about it here: http://lists.debian.org/debian-python/2006/01/msg00028.html -- Bob Tanner | Phone : (952)943-8700 http://www.real-time.com, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288 From cwmoad at gmail.com Thu Jan 19 19:23:20 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Thu, 19 Jan 2006 13:23:20 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> Message-ID: <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> Well here are th cvs links to them. http://cvs.sourceforge.net/viewcvs.py/matplotlib/matplotlib/setup.py?rev=1.143&view=auto http://cvs.sourceforge.net/viewcvs.py/matplotlib/toolkits/basemap/setup.py?rev=1.33&view=auto The matplotlib.toolkits module does not exist in the mpl cvs and it is not in the setup.py file. I have tried adding both. In basemap however, the matplotlib.toolkits module does exist and is listed in the setup.py file. On 1/19/06, Phillip J. Eby wrote: > At 11:44 AM 01/19/2006 -0500, Charlie Moad wrote: > >I tried the "." and I keep getting: > > > >error in basemap setup command: Distribution contains no modules or > >packages for namespace package 'matplotlib.toolkits' > > Great, now we're making progress. :) Can you post your setup.py? It > sounds like you don't have 'matplotlib.toolkits' listed in your 'packages' > keyword, or perhaps there's something amiss with your package_dir or something. > > From robert.kern at gmail.com Thu Jan 19 21:09:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 19 Jan 2006 14:09:06 -0600 Subject: [Distutils] namespace_packages In-Reply-To: <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> Message-ID: <43CFF1E2.60607@gmail.com> Charlie Moad wrote: > Well here are th cvs links to them. > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/matplotlib/setup.py?rev=1.143&view=auto > http://cvs.sourceforge.net/viewcvs.py/matplotlib/toolkits/basemap/setup.py?rev=1.33&view=auto > > The matplotlib.toolkits module does not exist in the mpl cvs and it is > not in the setup.py file. I have tried adding both. In basemap > however, the matplotlib.toolkits module does exist and is listed in > the setup.py file. Both of these setup.py files are buggy. The packages list needs to have entries in dotted form ("matplotlib.toolkits"), not filesystem form ("matplotlib/toolkits"). Correcting them, and adding an empty lib/matplotlib/toolkits/__init__.py to the matplotlib checkout allows me to build eggs with namespace_packages set appropriately. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From cwmoad at gmail.com Thu Jan 19 21:32:18 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Thu, 19 Jan 2006 15:32:18 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <43CFF1E2.60607@gmail.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> <43CFF1E2.60607@gmail.com> Message-ID: <6382066a0601191232o1b54f214xfad5383489a4b12c@mail.gmail.com> On 1/19/06, Robert Kern wrote: > Charlie Moad wrote: > > Well here are th cvs links to them. > > > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/matplotlib/setup.py?rev=1.143&view=auto > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/toolkits/basemap/setup.py?rev=1.33&view=auto > > > > The matplotlib.toolkits module does not exist in the mpl cvs and it is > > not in the setup.py file. I have tried adding both. In basemap > > however, the matplotlib.toolkits module does exist and is listed in > > the setup.py file. > > Both of these setup.py files are buggy. The packages list needs to have entries > in dotted form ("matplotlib.toolkits"), not filesystem form > ("matplotlib/toolkits"). Correcting them, and adding an empty > lib/matplotlib/toolkits/__init__.py to the matplotlib checkout allows me to > build eggs with namespace_packages set appropriately. > Thanks for taking a whack at this. I will test all your comments and commit if I get it working. From gh at ghaering.de Thu Jan 19 23:16:39 2006 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Thu, 19 Jan 2006 23:16:39 +0100 Subject: [Distutils] python setup.py test not working on Linux Message-ID: <43D00FC7.7060500@ghaering.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I tried adding the test command to my setuptools-enhanced build process. And I got "python setup.py test" to work on Windows, now when I'm on Linux it fails. I *think* that it was in both cases the latest version of setuptools. You should be able to verify it with these sources: svn co http://initd.org/svn/pysqlite/pysqlite/trunk/ pysqlite_trunk -r 202 Oh, you need SQLite version 3.2.2 or later to build this pysqlite project. This is the full output: gerhard at lara:~/src/svn/pysqlite/trunk$ python setup.py test running test running egg_info writing requirements to pysqlite.egg-info/requires.txt writing pysqlite.egg-info/PKG-INFO writing top-level names to pysqlite.egg-info/top_level.txt warning: manifest_maker: standard file not found: should have one of README, README.txt reading manifest template 'MANIFEST.in' writing manifest file 'pysqlite.egg-info/SOURCES.txt' running build_ext copying build/lib.linux-i686-2.4/pysqlite2/_sqlite.so -> lib Traceback (most recent call last): File "setup.py", line 151, in ? main() File "setup.py", line 137, in main classifiers = [ File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/test.py", line 59, in run self.run_tests() File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/setuptools/command/test.py", line 73, in run_tests unittest.main(None, None, [unittest.__file__]+self.test_args) File "/usr/lib/python2.4/unittest.py", line 758, in __init__ self.parseArgs(argv) File "/usr/lib/python2.4/unittest.py", line 785, in parseArgs self.createTests() File "/usr/lib/python2.4/unittest.py", line 791, in createTests self.module) File "/usr/lib/python2.4/unittest.py", line 556, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/usr/lib/python2.4/unittest.py", line 532, in loadTestsFromName parent, obj = obj, getattr(obj, part) AttributeError: 'module' object has no attribute 'test' - -- Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFD0A/HdIO4ozGCH14RAu1YAJkBQmM/QyuDvCbJEhQ71ggBsy3Y1QCghUee K2b9G/o1QIKqKLWUezRezQw= =RuvI -----END PGP SIGNATURE----- From pje at telecommunity.com Fri Jan 20 00:13:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 19 Jan 2006 18:13:43 -0500 Subject: [Distutils] python setup.py test not working on Linux In-Reply-To: <43D00FC7.7060500@ghaering.de> Message-ID: <5.1.1.6.0.20060119181051.025666e0@mail.telecommunity.com> At 11:16 PM 01/19/2006 +0100, Gerhard H?ring wrote: >AttributeError: 'module' object has no attribute 'test' This is probably an indication that the '.test' package was not importable for some reason. Try this: python setup.py develop python -c "import pysqlite2.test" and see what happens. The unittest package is rather notorious for not giving good error messages in the case of a failed import. It should really reraise the ImportError when it is unable to find an attribute. From chris at kateandchris.net Fri Jan 20 00:25:54 2006 From: chris at kateandchris.net (Chris Lambacher) Date: Thu, 19 Jan 2006 18:25:54 -0500 Subject: [Distutils] py2exe and setuptools w/ install_requires Message-ID: <20060119232554.GA13618@kateandchris.net> Hi, I have several apps that I distribute to customers using py2exe. I have also written modules which these apps share. Up to now I have not had any modules written in C so to build my py2exe apps I just set the PYTHONPATH environment variable to point to the module's location in CVS and run setup.py py2exe. This is convenient because release builds are done via Buildbot on a dedicated computer. All buildbot has to do is check out the source and run setup.py. I now have my first C module and have been trying to work out how to deal with building the module. I would prefer to build the module and 'release' it for use with my two apps. What I would like to do is build up an egg for the library and copy it to a known location. Then have the apps use the setuptools require_install directive to allow me to get the egg and 'install' it into the build directory somewhere and then let the rest of the py2exe magic work. I think I know where to add my call into setuptools from py2exe, but I cannot figure out what setup_tools function to call in order to do the 'installation'. Any help in executing my plan or in telling me I am a morron because there is some easier way to do this would be much appreciated. -Chris From gh at ghaering.de Fri Jan 20 01:35:18 2006 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Fri, 20 Jan 2006 01:35:18 +0100 Subject: [Distutils] python setup.py test not working on Linux In-Reply-To: <5.1.1.6.0.20060119181051.025666e0@mail.telecommunity.com> References: <5.1.1.6.0.20060119181051.025666e0@mail.telecommunity.com> Message-ID: <43D03046.5040103@ghaering.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote: > At 11:16 PM 01/19/2006 +0100, Gerhard H?ring wrote: > >> AttributeError: 'module' object has no attribute 'test' > > This is probably an indication that the '.test' package was not > importable for some reason. > > Try this: > > python setup.py develop > python -c "import pysqlite2.test" You were right! gerhard at lara:~/src/svn/pysqlite/trunk$ python -c "import pysqlite2.test" Traceback (most recent call last): File "", line 1, in ? File "/home/gerhard/src/svn/pysqlite/trunk/pysqlite2/test/__init__.py", line 25, in ? from pysqlite2.test import dbapi, types, userfunctions, factory, transactions File "/home/gerhard/src/svn/pysqlite/trunk/pysqlite2/test/dbapi.py", line 26, in ? import pysqlite2.dbapi2 as sqlite File "lib/dbapi2.py", line 32, in ? from pysqlite2._sqlite import * ImportError: pysqlite2/_sqlite.so: undefined symbol: sqlite3_transfer_bindings > and see what happens. The unittest package is rather notorious for not > giving good error messages in the case of a failed import. It should > really reraise the ImportError when it is unable to find an attribute. I forgot one step I usually always do when I get the above error message ... gerhard at lara:~/src/svn/pysqlite/trunk$ export LD_LIBRARY_PATH=/usr/local/lib/ gerhard at lara:~/src/svn/pysqlite/trunk$ python setup.py test -q running test running egg_info writing requirements to pysqlite.egg-info/requires.txt writing pysqlite.egg-info/PKG-INFO writing top-level names to pysqlite.egg-info/top_level.txt warning: manifest_maker: standard file not found: should have one of README, README.txt reading manifest template 'MANIFEST.in' writing manifest file 'pysqlite.egg-info/SOURCES.txt' running build_ext copying build/lib.linux-i686-2.4/pysqlite2/_sqlite.so -> lib ....................................................................................................................................................... - ---------------------------------------------------------------------- Ran 151 tests in 1.671s OK Now it works perfectly! Thanks. - -- Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD4DBQFD0DBGdIO4ozGCH14RAhwuAJ43ooK9MH7pTCiYVE1M9Vtbx0ZkywCYt597 S2lVHfrT3RpPtN0Uec44eQ== =muGK -----END PGP SIGNATURE----- From cwmoad at gmail.com Fri Jan 20 17:20:41 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Fri, 20 Jan 2006 11:20:41 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <43CFF1E2.60607@gmail.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> <43CFF1E2.60607@gmail.com> Message-ID: <6382066a0601200820y3c4f48fdv17a3e5b1ffb49f82@mail.gmail.com> On 1/19/06, Robert Kern wrote: > Charlie Moad wrote: > > Well here are th cvs links to them. > > > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/matplotlib/setup.py?rev=1.143&view=auto > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/toolkits/basemap/setup.py?rev=1.33&view=auto > > > > The matplotlib.toolkits module does not exist in the mpl cvs and it is > > not in the setup.py file. I have tried adding both. In basemap > > however, the matplotlib.toolkits module does exist and is listed in > > the setup.py file. > > Both of these setup.py files are buggy. The packages list needs to have entries > in dotted form ("matplotlib.toolkits"), not filesystem form > ("matplotlib/toolkits"). Correcting them, and adding an empty > lib/matplotlib/toolkits/__init__.py to the matplotlib checkout allows me to > build eggs with namespace_packages set appropriately. > I finally got time to try this, and it didn't help much. Changing all the packages to dot notation works fine. I have added the toolkits/__init__.py file. Setuptools warns about not having declare_namespace in the init files. I have added this: try: __import__('pkg_resources').declare_namespace(__name__) print __name__, __file__ except ImportError: pass # must not have setuptools to __init__.py in mpl's and basemap's matplotlib and matplotlib.toolkits module, hence in 4 places. Upon starting python I get this output: matplotlib /usr/lib/python2.4/site-packages/matplotlib-0.86.2cvs-py2.4-linux-i686.egg/matplotlib/__init__.pyc matplotlib.toolkits /usr/lib/python2.4/site-packages/matplotlib-0.86.2cvs-py2.4-linux-i686.egg/matplotlib/toolkits/__init__.pyc So the basemap declarations are never being reached. Where should I go from here? From john_sips_tea at yahoo.com Fri Jan 20 18:33:12 2006 From: john_sips_tea at yahoo.com (John M. Gabriele) Date: Fri, 20 Jan 2006 09:33:12 -0800 (PST) Subject: [Distutils] using setup.py to uninstall Message-ID: <20060120173312.78880.qmail@web80905.mail.scd.yahoo.com> Can distutils (setup.py) uninstall packages as well? If not, is this a feature that's being worked on, or is there some reason why it may not be necessary? Thanks, ---J __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From pje at telecommunity.com Fri Jan 20 18:31:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Jan 2006 12:31:12 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <6382066a0601200820y3c4f48fdv17a3e5b1ffb49f82@mail.gmail.co m> References: <43CFF1E2.60607@gmail.com> <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> <43CFF1E2.60607@gmail.com> Message-ID: <5.1.1.6.0.20060120122732.0251fec0@mail.telecommunity.com> At 11:20 AM 01/20/2006 -0500, Charlie Moad wrote: >On 1/19/06, Robert Kern wrote: > > Charlie Moad wrote: > > > Well here are th cvs links to them. > > > > > > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/matplotlib/setup.py?rev=1.143&view=auto > > > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/toolkits/basemap/setup.py?rev=1.33&view=auto > > > > > > The matplotlib.toolkits module does not exist in the mpl cvs and it is > > > not in the setup.py file. I have tried adding both. In basemap > > > however, the matplotlib.toolkits module does exist and is listed in > > > the setup.py file. > > > > Both of these setup.py files are buggy. The packages list needs to have > entries > > in dotted form ("matplotlib.toolkits"), not filesystem form > > ("matplotlib/toolkits"). Correcting them, and adding an empty > > lib/matplotlib/toolkits/__init__.py to the matplotlib checkout allows me to > > build eggs with namespace_packages set appropriately. > > > >I finally got time to try this, and it didn't help much. Changing all >the packages to dot notation works fine. I have added the >toolkits/__init__.py file. Setuptools warns about not having >declare_namespace in the init files. I have added this: > >try: > __import__('pkg_resources').declare_namespace(__name__) > print __name__, __file__ >except ImportError: > pass # must not have setuptools Note that if you don't have pkg_resources, namespace packages simply won't work, so there's little point in the try/except. >to __init__.py in mpl's and basemap's matplotlib and >matplotlib.toolkits module, hence in 4 places. >Upon starting python I get this output: > >matplotlib >/usr/lib/python2.4/site-packages/matplotlib-0.86.2cvs-py2.4-linux-i686.egg/matplotlib/__init__.pyc >matplotlib.toolkits >/usr/lib/python2.4/site-packages/matplotlib-0.86.2cvs-py2.4-linux-i686.egg/matplotlib/toolkits/__init__.pyc > >So the basemap declarations are never being reached. Where should I >go from here? Is the other egg you're looking for on sys.path? >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From pje at telecommunity.com Fri Jan 20 18:35:56 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Jan 2006 12:35:56 -0500 Subject: [Distutils] py2exe and setuptools w/ install_requires In-Reply-To: <20060119232554.GA13618@kateandchris.net> Message-ID: <5.1.1.6.0.20060120123146.0251cd70@mail.telecommunity.com> At 06:25 PM 01/19/2006 -0500, Chris Lambacher wrote: >Hi, > >I have several apps that I distribute to customers using py2exe. I have also >written modules which these apps share. Up to now I have not had any >modules written in C so to build my py2exe apps I just set the PYTHONPATH >environment variable to point to the module's location in CVS and run setup.py >py2exe. This is convenient because release builds are done via Buildbot on a >dedicated computer. All buildbot has to do is check out the source and run >setup.py. > >I now have my first C module and have been trying to work out how to deal with >building the module. I would prefer to build the module and 'release' it for >use with my two apps. > >What I would like to do is build up an egg for the library and copy it to a >known location. Then have the apps use the setuptools require_install >directive to allow me to get the egg and 'install' it into the build directory >somewhere and then let the rest of the py2exe magic work. > >I think I know where to add my call into setuptools from py2exe, but I cannot >figure out what setup_tools function to call in order to do the >'installation'. Run "setup.py develop" in the source tree of each project, starting with the dependencies and working your way up to the main application. Then run py2exe. See also the py2exe Wiki, which has a page of tips for using py2exe with eggs. However, if you have source checkouts for all the things your application includes, you don't need to worry about unzipping eggs. Note that this will *not* include egg metadata in your .exe build, so if you are using packages that require such metadata, they may not work correctly when embedded. From cwmoad at gmail.com Fri Jan 20 18:41:00 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Fri, 20 Jan 2006 12:41:00 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <5.1.1.6.0.20060120122732.0251fec0@mail.telecommunity.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> <43CFF1E2.60607@gmail.com> <5.1.1.6.0.20060120122732.0251fec0@mail.telecommunity.com> Message-ID: <6382066a0601200941t7d6f95u848063a6865feb9c@mail.gmail.com> On 1/20/06, Phillip J. Eby wrote: > At 11:20 AM 01/20/2006 -0500, Charlie Moad wrote: > >On 1/19/06, Robert Kern wrote: > > > Charlie Moad wrote: > > > > Well here are th cvs links to them. > > > > > > > > > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/matplotlib/setup.py?rev=1.143&view=auto > > > > > > http://cvs.sourceforge.net/viewcvs.py/matplotlib/toolkits/basemap/setup.py?rev=1.33&view=auto > > > > > > > > The matplotlib.toolkits module does not exist in the mpl cvs and it is > > > > not in the setup.py file. I have tried adding both. In basemap > > > > however, the matplotlib.toolkits module does exist and is listed in > > > > the setup.py file. > > > > > > Both of these setup.py files are buggy. The packages list needs to have > > entries > > > in dotted form ("matplotlib.toolkits"), not filesystem form > > > ("matplotlib/toolkits"). Correcting them, and adding an empty > > > lib/matplotlib/toolkits/__init__.py to the matplotlib checkout allows me to > > > build eggs with namespace_packages set appropriately. > > > > > > >I finally got time to try this, and it didn't help much. Changing all > >the packages to dot notation works fine. I have added the > >toolkits/__init__.py file. Setuptools warns about not having > >declare_namespace in the init files. I have added this: > > > >try: > > __import__('pkg_resources').declare_namespace(__name__) > > print __name__, __file__ > >except ImportError: > > pass # must not have setuptools > > Note that if you don't have pkg_resources, namespace packages simply won't > work, so there's little point in the try/except. I don't want an ImportError raised for people who don't have setuptools, which is definitely a majority of mpl users. > >to __init__.py in mpl's and basemap's matplotlib and > >matplotlib.toolkits module, hence in 4 places. > >Upon starting python I get this output: > > > >matplotlib > >/usr/lib/python2.4/site-packages/matplotlib-0.86.2cvs-py2.4-linux-i686.egg/matplotlib/__init__.pyc > >matplotlib.toolkits > >/usr/lib/python2.4/site-packages/matplotlib-0.86.2cvs-py2.4-linux-i686.egg/matplotlib/toolkits/__init__.pyc > > > >So the basemap declarations are never being reached. Where should I > >go from here? > > > Is the other egg you're looking for on sys.path? Yes. /usr/lib/python2.4/site-packages/basemap-0.8-py2.4-linux-i686.egg is in easy-install.pth. I can import its top level modules, just not the conflicting matplotlib.toolkits.basemap module. From cwmoad at gmail.com Fri Jan 20 18:44:00 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Fri, 20 Jan 2006 12:44:00 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <6382066a0601200941t7d6f95u848063a6865feb9c@mail.gmail.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> <43CFF1E2.60607@gmail.com> <5.1.1.6.0.20060120122732.0251fec0@mail.telecommunity.com> <6382066a0601200941t7d6f95u848063a6865feb9c@mail.gmail.com> Message-ID: <6382066a0601200944r6e390caeu50ab84580940310c@mail.gmail.com> Here are the eggs if you want to unzip and look at them. I installed both unzipped. http://euclid.uits.iupui.edu/~cmoad/mpleggs/matplotlib-0.86.2cvs-py2.4-linux-i686.egg http://euclid.uits.iupui.edu/~cmoad/mpleggs/basemap-0.8-py2.4-linux-i686.egg From bob at redivi.com Fri Jan 20 18:52:39 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 20 Jan 2006 09:52:39 -0800 Subject: [Distutils] using setup.py to uninstall In-Reply-To: <20060120173312.78880.qmail@web80905.mail.scd.yahoo.com> References: <20060120173312.78880.qmail@web80905.mail.scd.yahoo.com> Message-ID: On Jan 20, 2006, at 9:33 AM, John M. Gabriele wrote: > Can distutils (setup.py) uninstall packages as well? > > If not, is this a feature that's being worked on, or > is there some reason why it may not be necessary? distutils can't and nobody is working on it. setuptools probably could, but doesn't offer that as a feature (except to delete conflicting non-egg packages when installing an egg). -bob From pje at telecommunity.com Fri Jan 20 19:18:01 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Jan 2006 13:18:01 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <6382066a0601200944r6e390caeu50ab84580940310c@mail.gmail.co m> References: <6382066a0601200941t7d6f95u848063a6865feb9c@mail.gmail.com> <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> <43CFF1E2.60607@gmail.com> <5.1.1.6.0.20060120122732.0251fec0@mail.telecommunity.com> <6382066a0601200941t7d6f95u848063a6865feb9c@mail.gmail.com> Message-ID: <5.1.1.6.0.20060120131310.0255fc80@mail.telecommunity.com> At 12:44 PM 01/20/2006 -0500, Charlie Moad wrote: >Here are the eggs if you want to unzip and look at them. I installed >both unzipped. > >http://euclid.uits.iupui.edu/~cmoad/mpleggs/matplotlib-0.86.2cvs-py2.4-linux-i686.egg >http://euclid.uits.iupui.edu/~cmoad/mpleggs/basemap-0.8-py2.4-linux-i686.egg There are two issues that I saw. First, the matplotlib egg above does not declare a namespace in its matplotlib/__init__.py. Second, the basemap egg does not contain a matplotlib/__init__.py at all. Please note that you *must* include an __init__.py file for every namespace package and parent package thereof, in *every* egg that's part of the namespace package, and that these __init__.py files must all contain a namespace declaration. Missing even one can cause problems. For example, the basemap egg is simply not importable, because as far as Python is concerned it does not contain a matplotlib package (due to the missing __init__.py). From cwmoad at gmail.com Fri Jan 20 19:38:57 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Fri, 20 Jan 2006 13:38:57 -0500 Subject: [Distutils] namespace_packages In-Reply-To: <5.1.1.6.0.20060120131310.0255fc80@mail.telecommunity.com> References: <5.1.1.6.0.20060118150330.00ac86c0@mail.telecommunity.com> <5.1.1.6.0.20060119125148.0253b8c0@mail.telecommunity.com> <6382066a0601191023n2e80147aq573d2c4472294730@mail.gmail.com> <43CFF1E2.60607@gmail.com> <5.1.1.6.0.20060120122732.0251fec0@mail.telecommunity.com> <6382066a0601200941t7d6f95u848063a6865feb9c@mail.gmail.com> <5.1.1.6.0.20060120131310.0255fc80@mail.telecommunity.com> Message-ID: <6382066a0601201038u282eb409g2ecc55311937c9b9@mail.gmail.com> Thanks for all your help. It is finally working. Another commit pulled out my declare_nmsp in mpl/__init__.py and I needed to specify the matplotlib module in basemap's setup.py for the __init__.py to get included in the install. Thanks again, - Charlie On 1/20/06, Phillip J. Eby wrote: > At 12:44 PM 01/20/2006 -0500, Charlie Moad wrote: > >Here are the eggs if you want to unzip and look at them. I installed > >both unzipped. > > > >http://euclid.uits.iupui.edu/~cmoad/mpleggs/matplotlib-0.86.2cvs-py2.4-linux-i686.egg > >http://euclid.uits.iupui.edu/~cmoad/mpleggs/basemap-0.8-py2.4-linux-i686.egg > > There are two issues that I saw. First, the matplotlib egg above does not > declare a namespace in its matplotlib/__init__.py. Second, the basemap egg > does not contain a matplotlib/__init__.py at all. > > Please note that you *must* include an __init__.py file for every namespace > package and parent package thereof, in *every* egg that's part of the > namespace package, and that these __init__.py files must all contain a > namespace declaration. Missing even one can cause problems. For example, > the basemap egg is simply not importable, because as far as Python is > concerned it does not contain a matplotlib package (due to the missing > __init__.py). > > From tim.peters at gmail.com Fri Jan 20 20:18:40 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 20 Jan 2006 14:18:40 -0500 Subject: [Distutils] using setup.py to uninstall In-Reply-To: <20060120173312.78880.qmail@web80905.mail.scd.yahoo.com> References: <20060120173312.78880.qmail@web80905.mail.scd.yahoo.com> Message-ID: <1f7befae0601201118y7dab696buea8e1bd8c710d4d3@mail.gmail.com> [John M. Gabriele] > Can distutils (setup.py) uninstall packages as well? A Windows installer created by "setup.py bdist_wininst" creates the usual kind of uninstallation entry in the Windows Control Panel -> Add/Remove Programs applet. AFAIK that's unique to Windows, though. > If not, is this a feature that's being worked on, or > is there some reason why it may not be necessary? Unixheads are too macho to admit they need help uninstalling ;-) From chris at kateandchris.net Fri Jan 20 20:58:39 2006 From: chris at kateandchris.net (Chris Lambacher) Date: Fri, 20 Jan 2006 14:58:39 -0500 Subject: [Distutils] [Py2exe-users] Re: py2exe and setuptools w/ install_requires In-Reply-To: <5.1.1.6.0.20060120123146.0251cd70@mail.telecommunity.com> References: <20060119232554.GA13618@kateandchris.net> <5.1.1.6.0.20060120123146.0251cd70@mail.telecommunity.com> Message-ID: <20060120195839.GB4530@kateandchris.net> On Fri, Jan 20, 2006 at 12:35:56PM -0500, Phillip J. Eby wrote: > At 06:25 PM 01/19/2006 -0500, Chris Lambacher wrote: > >Hi, > > > >I have several apps that I distribute to customers using py2exe. I have > >also > >written modules which these apps share. Up to now I have not had any > >modules written in C so to build my py2exe apps I just set the PYTHONPATH > >environment variable to point to the module's location in CVS and run > >setup.py > >py2exe. This is convenient because release builds are done via Buildbot > >on a > >dedicated computer. All buildbot has to do is check out the source and run > >setup.py. > > > >I now have my first C module and have been trying to work out how to deal > >with > >building the module. I would prefer to build the module and 'release' it > >for > >use with my two apps. > > > >What I would like to do is build up an egg for the library and copy it to a > >known location. Then have the apps use the setuptools require_install > >directive to allow me to get the egg and 'install' it into the build > >directory > >somewhere and then let the rest of the py2exe magic work. > > > >I think I know where to add my call into setuptools from py2exe, but I > >cannot > >figure out what setup_tools function to call in order to do the > >'installation'. > > Run "setup.py develop" in the source tree of each project, starting with > the dependencies and working your way up to the main application. Then run > py2exe. > > See also the py2exe Wiki, which has a page of tips for using py2exe with > eggs. However, if you have source checkouts for all the things your > application includes, you don't need to worry about unzipping eggs. This is almost what I want, but I want to release my modules and have my app builds pick them up as eggs. When I build my module eggs, I publish them to a directory that an http server indexes. I then I have a build script that calls 'easy_install -m -Z -d tmpdir -f http://myserver/eggs module' for each module that I am interested in. Then I add all the new directories in tmpdir to my python path and call setup.py py2exe. The advantage of this is that each time I build I could potentially be using a different version of the module. I don't have to worry about not getting updates to that libarary because I am building a point release of my app on some strange branch. I wanted to make this action triggered by install_requires in the py2exe command, I am instead doing this externally before the py2exe command. > > Note that this will *not* include egg metadata in your .exe build, so if > you are using packages that require such metadata, they may not work > correctly when embedded. Not a problem for me, I am not that sofisticated yet :) Thanks for the help, Chris From seefeld at sympatico.ca Sat Jan 21 02:41:12 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Fri, 20 Jan 2006 20:41:12 -0500 Subject: [Distutils] expected non-zero return values when spawning subprocesses Message-ID: <43D19138.5020707@sympatico.ca> Hi there, I'm using distutils.spawn.spawn() to run subprocesses during my build commands. It now happens that one of those subprocesses will return a non-zero value that doesn't strictly correspond to an error, and thus I'd like to catch it from within my python code. It appears spawn() itself will raise a message containing a stringified version of the exit code, but not the exit code itself. This seems to imply that, in order to discriminate on the exit value, I have to catch DistutilsExecError, and then parse the string. This seems rather unelegant. Is there some better way ? Could the exception be modified to contain the original code (as an integral value) ? Thanks, Stefan From cwmoad at gmail.com Mon Jan 23 14:27:38 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Mon, 23 Jan 2006 08:27:38 -0500 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <43CF1451.1030600@colorstudy.com> References: <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> <5.1.1.6.0.20060117195446.024c0bb0@mail.telecommunity.com> <43CF1451.1030600@colorstudy.com> Message-ID: <6382066a0601230527u70b6e458o6b3ad6f087dd3bb@mail.gmail.com> On 1/18/06, Ian Bicking wrote: > Phillip J. Eby wrote: > > I checked what the NetBSD "pkgsrc" system does, and it uses the fact > > that there is a .dl.sourceforge.net subdomain for mirrors. I > > investigated further and found that dl.sourceforge.net is a round-robin > > (or random?) DNS for each of the mirrors in the subdomain, so there's a > > simple transformation from the user-visible download pages to the actual > > download address. You can see this in the new fix_sf_url() function I > > added to setuptools.package_index. > > I just got a Connection Refused error, but it worked on the second try. > As I remember, the port system typically has a set of links, and > frequently fails over from one link to the next, so I expect that this > error should be expected. When it is encountered, setuptools should > just try again until it works. I have noticed in the past that SF > mirrors go up and down quite frequently, probably too fast for DNS to > keep up. Does the pypi download url need to be updated to be compatible with svn setuptools? I am assuming no, but I am constantly getting 404's from easy_install, yet I can paste the dl.sf.net link into a browser and it works?! These files have been on sf long enough to be propgated around, and constantly ~= 20 attempts on the command line. I am seeing this with matplotlib, btw. Thanks, Charlie From pje at telecommunity.com Mon Jan 23 17:31:26 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jan 2006 11:31:26 -0500 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <6382066a0601230527u70b6e458o6b3ad6f087dd3bb@mail.gmail.com > References: <43CF1451.1030600@colorstudy.com> <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> <5.1.1.6.0.20060117195446.024c0bb0@mail.telecommunity.com> <43CF1451.1030600@colorstudy.com> Message-ID: <5.1.1.6.0.20060123105955.01e59680@mail.telecommunity.com> At 08:27 AM 1/23/2006 -0500, Charlie Moad wrote: >On 1/18/06, Ian Bicking wrote: > > Phillip J. Eby wrote: > > > I checked what the NetBSD "pkgsrc" system does, and it uses the fact > > > that there is a .dl.sourceforge.net subdomain for mirrors. I > > > investigated further and found that dl.sourceforge.net is a round-robin > > > (or random?) DNS for each of the mirrors in the subdomain, so there's a > > > simple transformation from the user-visible download pages to the actual > > > download address. You can see this in the new fix_sf_url() function I > > > added to setuptools.package_index. > > > > I just got a Connection Refused error, but it worked on the second try. > > As I remember, the port system typically has a set of links, and > > frequently fails over from one link to the next, so I expect that this > > error should be expected. When it is encountered, setuptools should > > just try again until it works. I have noticed in the past that SF > > mirrors go up and down quite frequently, probably too fast for DNS to > > keep up. > >Does the pypi download url need to be updated to be compatible with >svn setuptools? I am assuming no, but I am constantly getting 404's >from easy_install, yet I can paste the dl.sf.net link into a browser >and it works?! These files have been on sf long enough to be >propgated around, and constantly ~= 20 attempts on the command line. >I am seeing this with matplotlib, btw. Interesting. I just tried "easy_install matplotlib" and it worked the first time. I actually have never yet gotten a 404 or other problem using the dl.sf.net URLs on any package. However, a little experimentation shows that my Windows PC caches the DNS reply for an extended period, so I guess that means that if you get a bad mirror IP, you're stuck with it until the cache clears or you run "ipconfig /flushdns". (In contrast, my Linux machine returns a different IP every time the name is looked up.) A little experimentation with the socket module shows that I can get the full list of mirror IPs from Python, so I've changed setuptools in SVN to just randomly select one to use, which should fix the sticking problem on Windows (and any other platform where it occurs). It's not a perfect solution, since of course you can still end up with a bad mirror for some download, but I'm not sure what else to do, short of having some kind of option/configuration to control mirror selection. From ronaldoussoren at mac.com Mon Jan 23 17:39:25 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 23 Jan 2006 17:39:25 +0100 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <5.1.1.6.0.20060123105955.01e59680@mail.telecommunity.com> References: <43CF1451.1030600@colorstudy.com> <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> <5.1.1.6.0.20060117195446.024c0bb0@mail.telecommunity.com> <43CF1451.1030600@colorstudy.com> <5.1.1.6.0.20060123105955.01e59680@mail.telecommunity.com> Message-ID: <614D135D-4D0E-41FA-AFFB-112B90FEEAEB@mac.com> On 23-jan-2006, at 17:31, Phillip J. Eby wrote: > > A little experimentation with the socket module shows that I can > get the > full list of mirror IPs from Python, so I've changed setuptools in > SVN to > just randomly select one to use, which should fix the sticking > problem on > Windows (and any other platform where it occurs). > > It's not a perfect solution, since of course you can still end up > with a > bad mirror for some download, but I'm not sure what else to do, > short of > having some kind of option/configuration to control mirror selection. You could try several mirrors before giving up. I haven't looked at the sourcecode for setuptools yet, so I don't know if this is feaseable. Ronald > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From ianb at colorstudy.com Mon Jan 23 19:09:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 23 Jan 2006 12:09:46 -0600 Subject: [Distutils] Setting distutils options from sitecustomize Message-ID: <43D51BEA.4010403@colorstudy.com> Has anyone tried anything that involves setting the distutils options (e.g., where to install libraries) from sitecustomize or some other Python location? I want to put in logic that is more complex than can be expressed in a configuration file. Setuptools-specific is a-ok too. Monkeypatching distutils doesn't particularly bother me in this case either. Though that's not great either (if it goes in sitecustomize), since it means that there's an overhead to every Python startup to load distutils and patch it, even if distutils wouldn't have otherwise loaded. Jim Fulton asked about something similar on IRC too -- actually about having an easy_install binary that installed to a different location than normal (to the Zope instance path). Similar in that however such a script might work it might need to do similar things, setting distutils options at runtime. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From chris at trichech.us Mon Jan 23 19:48:00 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Mon, 23 Jan 2006 13:48:00 -0500 Subject: [Distutils] static linking in distutils/setuptools? Message-ID: Is there any way to specify static linking to particular libraries when building installers with distutils or setuptools? I want to be able to include the shared libraries that I am linking to in my builds. I did not see any info on this in the docs. Thanks, C. -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060123/36fa5f19/attachment.bin From cwmoad at gmail.com Mon Jan 23 20:01:02 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Mon, 23 Jan 2006 14:01:02 -0500 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: References: Message-ID: <6382066a0601231101k67150f40ib632092250995ab3@mail.gmail.com> With the OSX eggs I link and include static libraries for libpng and freetype. You can just put them in a directory and when building mpl use something like, "LDFLAGS=/tmp/static-libs python setup.py build". This will make setuptools look in the directory you specify first. On 1/23/06, Christopher Fonnesbeck wrote: > Is there any way to specify static linking to particular libraries > when building installers with distutils or setuptools? I want to be > able to include the shared libraries that I am linking to in my > builds. I did not see any info on this in the docs. > > Thanks, > C. > > -- > Christopher J. Fonnesbeck > > Population Ecologist, Marine Mammal Section > Fish & Wildlife Research Institute (FWC) > St. Petersburg, FL > > Adjunct Assistant Professor > Warnell School of Forest Resources > University of Georgia > Athens, GA > > T: 727.235.5570 > E: chris at trichech.us > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > > > From ronaldoussoren at mac.com Mon Jan 23 20:16:43 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 23 Jan 2006 20:16:43 +0100 Subject: [Distutils] Setting distutils options from sitecustomize In-Reply-To: <43D51BEA.4010403@colorstudy.com> References: <43D51BEA.4010403@colorstudy.com> Message-ID: <80E746C5-C741-41F0-BEDF-B6834DA9B5FC@mac.com> On 23-jan-2006, at 19:09, Ian Bicking wrote: > Has anyone tried anything that involves setting the distutils options > (e.g., where to install libraries) from sitecustomize or some other > Python location? I want to put in logic that is more complex than can > be expressed in a configuration file. Setuptools-specific is a-ok > too. > Monkeypatching distutils doesn't particularly bother me in this case > either. Though that's not great either (if it goes in sitecustomize), > since it means that there's an overhead to every Python startup to > load > distutils and patch it, even if distutils wouldn't have otherwise > loaded. You could monkeypatch using .pth tricks. If you do this correctly the patch would only affect distutils. I've never used this in production, but something like this would work: - Create a directory (/FOOBAR) containing a directory distutils and an __init__.py inside that. The __init__.py should set its __path__ to include the real distutils package, execfile the real __init__.py and then do as much monkeypatching as it likes - Add a distutils-hacks.pth to site-packages that contains: import sys; sys.path.insert(0, "/FOOBAR") That way imports of distutils will get the monkeypatched version, without automagically importing distutils for each and every script. Extending distutils or setuptools to do what you need would probably be a much better solution :-) Ronald > > Jim Fulton asked about something similar on IRC too -- actually about > having an easy_install binary that installed to a different location > than normal (to the Zope instance path). Similar in that however > such a > script might work it might need to do similar things, setting > distutils > options at runtime. > > -- > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From matt at pollenation.net Tue Jan 24 01:42:58 2006 From: matt at pollenation.net (Matt Goodall) Date: Tue, 24 Jan 2006 00:42:58 -0000 (GMT) Subject: [Distutils] no access to resources unpacked as root Message-ID: <43054.82.0.168.223.1138063378.squirrel@webmail.pollenation.net> Hi, I'm using eggs a fair amount to deploy Twisted-based server applications. (This problem is not specific to Twisted so please don't stop reading!) The Twisted process starts up as root to open low ports, etc and then, once up and running, sheds privileges and continues to run as a normal user. Many of the resources that are buried in the eggs are located with resource_filename at module import time; some when the application is still running as root. The resources are unpacked from the egg into /root/.python-eggs/ and (correctly, I think) given 600 (rw-------) file access permissions. The problem is that the user the application ends up running as cannot access the unpacked resource files and fails with a PermissionDenied error. Now, I can move some uses of resource_filename to happens later but I don't think I can replace all of them without changing APIs. Has anyone else tackled this problem or can anyone offer any advice on how to avoid the problem in the first place? Thanks in advance, - Matt From pje at telecommunity.com Tue Jan 24 02:18:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Jan 2006 20:18:10 -0500 Subject: [Distutils] no access to resources unpacked as root In-Reply-To: <43054.82.0.168.223.1138063378.squirrel@webmail.pollenation .net> Message-ID: <5.1.1.6.0.20060123201716.01e4a418@mail.telecommunity.com> At 12:42 AM 1/24/2006 +0000, Matt Goodall wrote: >The Twisted process starts up as root to open low ports, etc and then, >once up and running, sheds privileges and continues to run as a normal >user. Many of the resources that are buried in the eggs are located with >resource_filename at module import time; some when the application is >still running as root. > >The resources are unpacked from the egg into /root/.python-eggs/ and >(correctly, I think) given 600 (rw-------) file access permissions. The >problem is that the user the application ends up running as cannot access >the unpacked resource files and fails with a PermissionDenied error. > >Now, I can move some uses of resource_filename to happens later but I >don't think I can replace all of them without changing APIs. > >Has anyone else tackled this problem or can anyone offer any advice on how >to avoid the problem in the first place? The simplest approach is probably to mark the eggs as not zip-safe, or to configure easy_install to install them unzipped. From strawman at astraw.com Tue Jan 24 08:50:21 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 23 Jan 2006 23:50:21 -0800 Subject: [Distutils] (extension) module dependencies in setuptools Message-ID: <43D5DC3D.7060002@astraw.com> I would like to propose a feature for setuptools: runtime enforcement of dependencies specified at build time by setup.py. I appreciate that "pkg_resources.require('foo==1.0')" works, but this requires a tedious update of version numbers in affected source files every time you upgrade foo and rebuild the target package. I'm thinking, in particular, of extension modules built on a particular version of another package with its own C interface. Think matplotlib.backends._ns_backend_agg depending on numpy. It would be really nice, in matplotlib's setup.py file, to say something like: from setuptools import setup, Extension import numpy numpy_include_dirs = numpy.get_numpy_include() setup(name='matplotlib', ext_modules=Extension('matplotlib.backends._ns_backend_agg', sources='src/backends/_ns_backend_agg.cpp', include_dirs=numpy_include_dirs, runtime_requires=[pkg_resources.get_distribution('numpy').as_requirement()], ) ) Alternatively, the whole package (not just the extension module) might depend on a particular version: from setuptools import setup import numpy numpy_include_dirs = numpy.get_numpy_include() setup(name='matplotlib', ext_modules=Extension('matplotlib.backends._ns_backend_agg', sources='src/backends/_ns_backend_agg.cpp', include_dirs=numpy_include_dirs, ) runtime_requires=[pkg_resources.get_distribution('numpy').as_requirement()], ) Now, if I installed an additional, newer numpy, a couple of things could happen: if my application imports matplotlib first, setuptools puts the appropriate (older) numpy into the global working_set and I get this older numpy when I do "import numpy". If my application imports (the newer) numpy first and then matplotlib, an exception is raised saying that matplotlib depends of versions such and such but version so and so is already imported. Because I'm thinking primarily of extension modules, there are additional reasons why I don't want to specify a runtime check using a hardcoded pkg_resources.require() in the package itself. First, the actual requirement may be a C-interface issue leading to segfaults and other nastiness if ignored or forgotten, thus justifying this easier way to specify dependencies. Second, an extension module, by definition, is not Python, so it will take more programming effort to write the call to pkg_resources.require(). Third, I really don't want to have to convince all the projects out there to modify multiple files to use setuptools. I'm already attempting to dispel enough anti-egg sentiment (for reasons I don't understand) resulting from slight changes to setup.py. What do folks think about this idea? Would such a feature be possible and desirable in setuptools? Cheers! Andrew From bbangert at groovie.org Wed Jan 25 00:54:48 2006 From: bbangert at groovie.org (bbangert at groovie.org) Date: Tue, 24 Jan 2006 15:54:48 -0800 (PST) Subject: [Distutils] setuptools include_package_data ignore data under a 'package'? Message-ID: <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> I've been having some strange issues when trying to get include_package_data to actually include package data, when said data is under a module. Though it seems to behave unpredictably. For example, several people have installed Pylons and its been missing pylons/media/style/orange.css (a package file), though I haven't had this happen myself. However, when I make a new template project, then bundle it into a distribution and unpack it (to see what setuptools thought was package data). It's missing several directories. Easy way to reproduce: paster create --template=pylons test cd test python setup.py sdist cd dist tar xzvf test-0.0.0dev.tar.gz Then compare: ls test-0.0.0dev/test/ To ../test/ The new one is missing the docs, public, templates directories which are package data directories containing content. If these aren't included on purpose, where can I get more information on exactly what include_package_data=True does actually include? BTW, I'm using setuptools 0.6a9 Thanks, Ben From bbangert at groovie.org Wed Jan 25 01:33:19 2006 From: bbangert at groovie.org (bbangert at groovie.org) Date: Tue, 24 Jan 2006 16:33:19 -0800 (PST) Subject: [Distutils] setuptools include_package_data ignore data under a 'package'? In-Reply-To: <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> References: <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> Message-ID: <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> > The new one is missing the docs, public, templates directories which are > package data directories containing content. If these aren't included on > purpose, where can I get more information on exactly what > include_package_data=True does actually include? > > BTW, I'm using setuptools 0.6a9 Nevermind, I already see this was due to me not having checked in the project to a version control system or registering all the files manually in MANIFEST.in. Any plans to support darcs? - Ben From bob at redivi.com Wed Jan 25 02:08:48 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 24 Jan 2006 17:08:48 -0800 Subject: [Distutils] setuptools include_package_data ignore data under a 'package'? In-Reply-To: <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> References: <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> Message-ID: <2C618C89-2038-4594-B2FD-FFE0F5D1B2CC@redivi.com> On Jan 24, 2006, at 4:33 PM, bbangert at groovie.org wrote: >> The new one is missing the docs, public, templates directories >> which are >> package data directories containing content. If these aren't >> included on >> purpose, where can I get more information on exactly what >> include_package_data=True does actually include? >> >> BTW, I'm using setuptools 0.6a9 > > Nevermind, I already see this was due to me not having checked in the > project to a version control system or registering all the files > manually > in MANIFEST.in. > > Any plans to support darcs? If you want darcs (hg, svk, perforce, ...) support, I'm sure it'd get added if you wrote a patch :) Perhaps setuptools should support manifest generating hooks via an entry point? -bob From pje at telecommunity.com Wed Jan 25 02:32:46 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jan 2006 20:32:46 -0500 Subject: [Distutils] setuptools include_package_data ignore data under a 'package'? In-Reply-To: <2C618C89-2038-4594-B2FD-FFE0F5D1B2CC@redivi.com> References: <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> Message-ID: <5.1.1.6.0.20060124202958.022c2360@mail.telecommunity.com> At 05:08 PM 1/24/2006 -0800, Bob Ippolito wrote: >On Jan 24, 2006, at 4:33 PM, bbangert at groovie.org wrote: > > >> The new one is missing the docs, public, templates directories > >> which are > >> package data directories containing content. If these aren't > >> included on > >> purpose, where can I get more information on exactly what > >> include_package_data=True does actually include? > >> > >> BTW, I'm using setuptools 0.6a9 > > > > Nevermind, I already see this was due to me not having checked in the > > project to a version control system or registering all the files > > manually > > in MANIFEST.in. > > > > Any plans to support darcs? > >If you want darcs (hg, svk, perforce, ...) support, I'm sure it'd get >added if you wrote a patch :) > >Perhaps setuptools should support manifest generating hooks via an >entry point? Yeah, this is actually on my list, just haven't got 'round to it yet. For systems that include their control data in-band (as files/dirs in the project area) it's pretty easy, but my impression was that most things besides CVS and Subversion don't do this. It would help if folks using the other tools could give some idea of how you go about finding tracked files using them, so I could get a better idea of how the extension API should work. From trentm at ActiveState.com Wed Jan 25 03:14:15 2006 From: trentm at ActiveState.com (Trent Mick) Date: Tue, 24 Jan 2006 18:14:15 -0800 Subject: [Distutils] setuptools include_package_data ignore data under a 'package'? In-Reply-To: <5.1.1.6.0.20060124202958.022c2360@mail.telecommunity.com> References: <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> <5.1.1.6.0.20060124202958.022c2360@mail.telecommunity.com> Message-ID: <20060125021415.GA1229@activestate.com> [Phillip J. Eby wrote] > Yeah, this is actually on my list, just haven't got 'round to it yet. For > systems that include their control data in-band (as files/dirs in the > project area) it's pretty easy, but my impression was that most things > besides CVS and Subversion don't do this. It would help if folks using the > other tools could give some idea of how you go about finding tracked files > using them, so I could get a better idea of how the extension API should work. If I understand what you are asking here, you get the list of Perforce files in the current client view (kinda the equivalent of an SVN working copy) with the "p4 have ./..." command: $ p4 help have have -- List revisions last synced p4 have [ file ... ] List revisions of named files that were last synced from the depot. If no file name is given list all files synced on this client. The format is depot-file#revision - client-file $ p4 have ./... //depot/main/Apps/ActivePython-devel/README.txt#9 - c:\trentm\as\ActivePython-devel\README.txt ... Some notes: - Why "./..."? Firstly, you need to specify a path argument because otherwise p4 defaults to the *whole* client view, which may (and likely does) include directories outside of your current dir. Secondly, "..." is Perforce's way of say "recursively under the given directory". So "./..." as a path argument effectively makes a p4 command behave like an svn/cvs command's default: recursively under the current dir. - If you *do* get into the business of doing this Perforce stuff, you may want to steal from or use my "p4lib.py" (either of which you are welcome todo -- MIT license): http://trentm.com/projects/px/p4lib.py which provides parsing of some of the common p4 commands. (It is not perfect and arguably the Right Thing for Python/Perforce bindings would use Perforce's C++ API. 'p4lib.py' does have the benefit of being pure Python.) >>> import p4lib, sys, pprint >>> sys.displayhook = pprint.pprint >>> p4 = p4lib.P4() >>> p4.have("./...") [{'depotFile': '//depot/main/Apps/ActivePython-devel/README.txt', 'localFile': 'c:\\trentm\\as\\ActivePython-devel\\README.txt', 'rev': 9}, ... ] Cheers, Trent -- Trent Mick TrentM at ActiveState.com From pje at telecommunity.com Wed Jan 25 05:14:55 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Jan 2006 23:14:55 -0500 Subject: [Distutils] setuptools include_package_data ignore data under a 'package'? In-Reply-To: <20060125021415.GA1229@activestate.com> References: <5.1.1.6.0.20060124202958.022c2360@mail.telecommunity.com> <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> <5.1.1.6.0.20060124202958.022c2360@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060124230840.022c4190@mail.telecommunity.com> At 06:14 PM 1/24/2006 -0800, Trent Mick wrote: >[Phillip J. Eby wrote] > > Yeah, this is actually on my list, just haven't got 'round to it yet. For > > systems that include their control data in-band (as files/dirs in the > > project area) it's pretty easy, but my impression was that most things > > besides CVS and Subversion don't do this. It would help if folks using > the > > other tools could give some idea of how you go about finding tracked files > > using them, so I could get a better idea of how the extension API > should work. > >If I understand what you are asking here, you get the list of Perforce >files in the current client view (kinda the equivalent of an SVN working >copy) with the "p4 have ./..." command: >[snip] Actually, I was looking more for how to identify this sort of thing without searching the system to see where the executables are, and then trying to figure out if 'p4' is really Perforce or not. Subversion and CVS are pretty unambiguous to detect; I was hoping for something equally unambiguous for other systems. Certainly, to implement revision control support in the actual setuptools core code, it needs to be doable without requiring anything outside the standard library. Plugins to support revision control don't have this issue, and they're needed only by the original developer making source distributions from checkouts. So if there's a plugin to implement say, Perforce support, it's okay for that plugin to have library dependencies or require special configuration files or whatever, so long as they can be found via environment variables or via distutils configuration options. I think what I should probably do at this point is just refactor the current file finder to be a plugin itself, and publish the API. Then, I guess people wanting to implement support for other systems can tell me if the API is lacking something they need. That'll probably be easier than trying to design a plugin API from scratch. From mstarzyk at gmail.com Wed Jan 25 14:53:31 2006 From: mstarzyk at gmail.com (Maciek Starzyk) Date: Wed, 25 Jan 2006 14:53:31 +0100 Subject: [Distutils] Problem upgrading docutils with setuptools Message-ID: <7a13e1e0601250553w31bb8697l6e0732014cbac654@mail.gmail.com> Hi. Recently I had some troubles with clean update of docutils. 0.3.9 -> 0.4 It looks to me like a setuptools problem. See below: First install docutils 0.3.9: ----------------------- 8< ---------------------------------- D:\download\python\docutils>easy_install docutils-0.3.9.tar.gz Processing docutils-0.3.9.tar.gz Running docutils-0.3.9\setup.py -q bdist_egg --dist-dir c:\docume~1\mast\ustawi~1\temp\easy_install-cxyfp2\docutils-0.3.9\egg-dist-tmp-rcp8-e "optparse" module already present; ignoring extras/optparse.py. "textwrap" module already present; ignoring extras/textwrap.py. zip_safe flag not set; analyzing archive contents... Adding docutils 0.3.9 to easy-install.pth file Installing rst2latex.py script to C:\Python24\Scripts Installing rst2html.py script to C:\Python24\Scripts Installed c:\python24\lib\site-packages\docutils-0.3.9-py2.4.egg Processing dependencies for docutils==0.3.9 D:\download\python\docutils>python -c "import docutils.parsers.rst; print 'OK'" OK ----------------------- 8< ---------------------------------- Then upgrade to docutils 0.4. Note that setup says "roman" module is already there. It was installed as a part of docutils 0.3.9. At the moment when setup checks for module roman - docutils 0.3.9 egg is still on the PYTHONPATH and therefore "roman" is not installed. But later on docutils 0.3.9 egg is removed from PYTHONPATH and docutils 0.4 fails: ----------------------- 8< ---------------------------------- D:\download\python\docutils>easy_install -U docutils-0.4.tar.gz Processing docutils-0.4.tar.gz Running docutils-0.4\setup.py -q bdist_egg --dist-dir c:\docume~1\mast\ustawi~1\ temp\easy_install-hnciga\docutils-0.4\egg-dist-tmp-y4g2fn "optparse" module already present; ignoring extras/optparse.py. "textwrap" module already present; ignoring extras/textwrap.py. "roman" module already present; ignoring extras/roman.py. zip_safe flag not set; analyzing archive contents... docutils.parsers.rst.directives.misc: module references __file__ docutils.writers.html4css1.__init__: module references __file__ docutils.writers.newlatex2e.__init__: module references __file__ docutils.writers.pep_html.__init__: module references __file__ docutils.writers.s5_html.__init__: module references __file__ Removing docutils 0.3.9 from easy-install.pth file Adding docutils 0.4 to easy-install.pth file Installing rst2html.py script to C:\Python24\Scripts Installing rst2latex.py script to C:\Python24\Scripts Installing rst2newlatex.py script to C:\Python24\Scripts Installing rst2pseudoxml.py script to C:\Python24\Scripts Installing rst2s5.py script to C:\Python24\Scripts Installing rst2xml.py script to C:\Python24\Scripts Installed c:\python24\lib\site-packages\docutils-0.4-py2.4.egg Processing dependencies for docutils==0.4 D:\download\python\docutils>python -c "import docutils.parsers.rst; print 'OK'" Traceback (most recent call last): File "", line 1, in ? File "c:\python24\lib\site-packages\docutils-0.4-py2.4.egg\docutils\parsers\rst\__init__.py", line 77, in ? from docutils.parsers.rst import states File "c:\python24\lib\site-packages\docutils-0.4-py2.4.egg\docutils\parsers\rst\states.py", line 110, in ? import roman ImportError: No module named roman ----------------------- 8< ---------------------------------- EasyInstall docs say: > installing a package automatically replaces > any previous version in the easy-install.pth file So maybe this replacing should be done earlier - ensuring that the previous version's code is not in path - before actually running setup ? Or is it a bug in docutils' setup.py ? Cheers, Maciek From trentm at ActiveState.com Wed Jan 25 18:22:26 2006 From: trentm at ActiveState.com (Trent Mick) Date: Wed, 25 Jan 2006 09:22:26 -0800 Subject: [Distutils] setuptools include_package_data ignore data under a 'package'? In-Reply-To: <5.1.1.6.0.20060124230840.022c4190@mail.telecommunity.com> References: <5.1.1.6.0.20060124202958.022c2360@mail.telecommunity.com> <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> <52061.69.107.93.194.1138146888.squirrel@69.107.93.194> <52200.69.107.93.194.1138149199.squirrel@69.107.93.194> <5.1.1.6.0.20060124202958.022c2360@mail.telecommunity.com> <5.1.1.6.0.20060124230840.022c4190@mail.telecommunity.com> Message-ID: <20060125172226.GB19879@activestate.com> [Phillip J. Eby wrote] > Actually, I was looking more for how to identify this sort of thing without > searching the system to see where the executables are, and then trying to > figure out if 'p4' is really Perforce or not. Subversion and CVS are > pretty unambiguous to detect; I was hoping for something equally > unambiguous for other systems. Certainly, to implement revision control > support in the actual setuptools core code, it needs to be doable without > requiring anything outside the standard library. How are you getting the list of files in a subversion working copy if not by running svn. Are you using the .svn control dirs? Are the entries file formats publicly speced and guaranteed to stay backward compatible? > I think what I should probably do at this point is just refactor the > current file finder to be a plugin itself, and publish the API. Anyway, yah an API for this would probably be good. Cheers, Trent -- Trent Mick TrentM at ActiveState.com From pje at telecommunity.com Wed Jan 25 18:31:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jan 2006 12:31:12 -0500 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <7a13e1e0601250553w31bb8697l6e0732014cbac654@mail.gmail.com > Message-ID: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> At 02:53 PM 1/25/2006 +0100, Maciek Starzyk wrote: >Then upgrade to docutils 0.4. >Note that setup says "roman" module is already there. It was installed >as a part of docutils 0.3.9. At the moment when setup checks for >module roman - docutils 0.3.9 egg is still on the PYTHONPATH and therefore >"roman" is not installed. A simple fix is to do "easy_install -m docutils" before the upgrade, as this will remove the existing version from the .pth file. > But later on docutils 0.3.9 egg is removed from >PYTHONPATH and docutils 0.4 fails: > >EasyInstall docs say: > > installing a package automatically replaces > > any previous version in the easy-install.pth file > >So maybe this replacing should be done earlier - ensuring that the >previous version's code >is not in path - before actually running setup ? > >Or is it a bug in docutils' setup.py ? Well, docutils' setup.py is clearly trying to support Python versions older than 2.3, or else it would not be checking for textwrap and optparse. Setuptools only supports 2.3 and higher. So those modules shouldn't be an issue. If docutils were intended to use setuptools, then it would not make sense to check for 'roman', either, since an external dependency should be just that, a dependency. So in that case I would consider it a bug in their setup.py. However, since they clearly are trying to support 2.2 and maybe older versions as well, it's hard to call it that. Rather, I'd have to say it's a limitation of setuptools' ability to be backward-compatible with highly-customized setup scripts. Unfortunately, there is no way to remove the existing version from sys.path before the setup script runs, as before setup() is called, there's no way to know precisely which project needs to be removed. EasyInstall accepts URLs and directory names as well as project names, so it's not guaranteed that it will know what project it's building. EasyInstall also doesn't change sys.path at all under normal circumstances; it only changes the .pth file so that sys.path will be changed the next time Python is started. If I were to propose a change to docutils' setup.py, I would suggest that roman.py should either be bundled as part of the docutils package (i.e. always installed, but inside the docutils package directory), or else be treated as an external dependency (never installed by the setup script directly, only by the user or by setuptools' dependency handling). Either of these approaches is a robust way to deal with the issue. Checking for modules' presence at installation time, however, isn't a good practice at all; I found it to be fragile and error-prone for PEAK (which used to rely on zope.interface, among other things) before setuptools even existed. From mstarzyk at gmail.com Wed Jan 25 18:46:57 2006 From: mstarzyk at gmail.com (Maciek Starzyk) Date: Wed, 25 Jan 2006 18:46:57 +0100 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> Message-ID: <7a13e1e0601250946m22976d5dq511cb74bec447ae3@mail.gmail.com> Hi Phillip, Thanks for your response. On 1/25/06, Phillip J. Eby wrote: > A simple fix is to do "easy_install -m docutils" before the upgrade, as > this will remove the existing version from the .pth file. Actually this problem was triggered by installing my package that depends on docutils. In setup.py this package has: setup( ... install_requires = ['docutils>=0.4'] ) Is there a way to tell setup.py to do the equivalent of easy_install -m while upgrading its dependencies ? Cheers, Maciek From bob at redivi.com Wed Jan 25 18:48:03 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 25 Jan 2006 09:48:03 -0800 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> Message-ID: <6760D25D-6ECB-4A0D-8544-C3A60B8EBC36@redivi.com> On Jan 25, 2006, at 9:31 AM, Phillip J. Eby wrote: > At 02:53 PM 1/25/2006 +0100, Maciek Starzyk wrote: >> Then upgrade to docutils 0.4. >> Note that setup says "roman" module is already there. It was >> installed >> as a part of docutils 0.3.9. At the moment when setup checks for >> module roman - docutils 0.3.9 egg is still on the PYTHONPATH and >> therefore >> "roman" is not installed. > > A simple fix is to do "easy_install -m docutils" before the > upgrade, as > this will remove the existing version from the .pth file. Another simple fix is just to run easy_install docutils a second time if you had it previously installed. Its setup.py script is stupid and fragile. -bob From pje at telecommunity.com Wed Jan 25 19:11:27 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jan 2006 13:11:27 -0500 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <6760D25D-6ECB-4A0D-8544-C3A60B8EBC36@redivi.com> References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060125130602.022c2048@mail.telecommunity.com> At 09:48 AM 1/25/2006 -0800, Bob Ippolito wrote: >Another simple fix is just to run easy_install docutils a second time >if you had it previously installed. Its setup.py script is stupid >and fragile. Geez, dude, I already said that in a diplomatic way, i.e., without saying "stupid" and without saying *their* setup was fragile. They're just people like you and me who want their stuff to work; they don't deserve to have their work called stupid. I wasn't stupid when I tried doing that kind of check-for-and-install dependency installation; it's just the simplest thing that could possibly work. The fact that it doesn't actually work all that well is something to learn from, and trying stuff in order to learn what works doesn't make you stupid - it makes you smarter. From pje at telecommunity.com Wed Jan 25 19:14:13 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jan 2006 13:14:13 -0500 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <7a13e1e0601250946m22976d5dq511cb74bec447ae3@mail.gmail.com > References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060125131201.023e3370@mail.telecommunity.com> At 06:46 PM 1/25/2006 +0100, Maciek Starzyk wrote: >Is there a way to tell setup.py to do the equivalent of easy_install -m >while upgrading its dependencies ? No, not really. You could always just bundle the "roman" module in your project, I suppose, or create a package for it on PyPI and make it a requirement of your project. These approaches wouldn't change docutils' behavior, but it'd ensure that *your* projects using docutils would work. From bob at redivi.com Wed Jan 25 19:23:56 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 25 Jan 2006 10:23:56 -0800 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <5.1.1.6.0.20060125130602.022c2048@mail.telecommunity.com> References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125130602.022c2048@mail.telecommunity.com> Message-ID: <7AC1318E-16DA-4AC1-85CF-6F8CAEE2B981@redivi.com> On Jan 25, 2006, at 10:11 AM, Phillip J. Eby wrote: > At 09:48 AM 1/25/2006 -0800, Bob Ippolito wrote: >> Another simple fix is just to run easy_install docutils a second time >> if you had it previously installed. Its setup.py script is stupid >> and fragile. > > Geez, dude, I already said that in a diplomatic way, i.e., without > saying "stupid" and without saying *their* setup was fragile. > They're just people like you and me who want their stuff to work; > they don't deserve to have their work called stupid. Well, I've had it break on me even without setuptools.. it's annoying. I didn't mean any offense to the author. -bob From bob at redivi.com Wed Jan 25 19:26:49 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 25 Jan 2006 10:26:49 -0800 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <5.1.1.6.0.20060125131201.023e3370@mail.telecommunity.com> References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125131201.023e3370@mail.telecommunity.com> Message-ID: On Jan 25, 2006, at 10:14 AM, Phillip J. Eby wrote: > At 06:46 PM 1/25/2006 +0100, Maciek Starzyk wrote: >> Is there a way to tell setup.py to do the equivalent of >> easy_install -m >> while upgrading its dependencies ? > > No, not really. You could always just bundle the "roman" module in > your > project, I suppose, or create a package for it on PyPI and make it a > requirement of your project. These approaches wouldn't change > docutils' > behavior, but it'd ensure that *your* projects using docutils would > work. Would setuptools notice the conflict between the roman module in both eggs? -bob From pje at telecommunity.com Wed Jan 25 19:36:51 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jan 2006 13:36:51 -0500 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: References: <5.1.1.6.0.20060125131201.023e3370@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125131201.023e3370@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060125133449.02096b18@mail.telecommunity.com> At 10:26 AM 1/25/2006 -0800, Bob Ippolito wrote: >On Jan 25, 2006, at 10:14 AM, Phillip J. Eby wrote: > >>At 06:46 PM 1/25/2006 +0100, Maciek Starzyk wrote: >>>Is there a way to tell setup.py to do the equivalent of >>>easy_install -m >>>while upgrading its dependencies ? >> >>No, not really. You could always just bundle the "roman" module in >>your >>project, I suppose, or create a package for it on PyPI and make it a >>requirement of your project. These approaches wouldn't change >>docutils' >>behavior, but it'd ensure that *your* projects using docutils would >>work. > >Would setuptools notice the conflict between the roman module in both >eggs? Not as long as docutils is installed via EasyInstall. Setuptools doesn't care about duplication *inside* eggs, only with stuff installed *outside* of eggs. (Unfortunately, there's a bit of an issue with .egg-info eggs, in that you still get conflict warnings for them at the moment.) From smurf at smurf.noris.de Wed Jan 25 19:44:34 2006 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Wed, 25 Jan 2006 19:44:34 +0100 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125131201.023e3370@mail.telecommunity.com> Message-ID: <20060125184434.GD26418@kiste.smurf.noris.de> Hi, Bob Ippolito: > Would setuptools notice the conflict between the roman module in both > eggs? > There is no conflict -- your module would be called "whatever.roman". -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - "You can't find a hermit to teach you herming, because of course that rather spoils the whole thing." -- Terry Pratchett (Small Gods) From bob at redivi.com Wed Jan 25 20:11:04 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 25 Jan 2006 11:11:04 -0800 Subject: [Distutils] Problem upgrading docutils with setuptools In-Reply-To: <20060125184434.GD26418@kiste.smurf.noris.de> References: <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125121939.04132008@mail.telecommunity.com> <5.1.1.6.0.20060125131201.023e3370@mail.telecommunity.com> <20060125184434.GD26418@kiste.smurf.noris.de> Message-ID: On Jan 25, 2006, at 10:44 AM, Matthias Urlichs wrote: > Bob Ippolito: >> Would setuptools notice the conflict between the roman module in both >> eggs? >> > There is no conflict -- your module would be called "whatever.roman". No, there is. Including roman was suggested as a way to satisfy a docutils dependency. Docutils expects roman to be a top-level module, and it will install it -- but only if roman can't be imported at the time setup.py is run. This behavior is problematic for packaging software (eggs and any other bdist command). Basically, if the person packaging docutils happens to have docutils installed at the time of packaging, the resultant package will be missing a required component. In the case of eggs, the detected roman module will disappear as soon as the new egg is activated, so you have to easy_install it twice (or equivalent) in order to get something that works. -bob From seefeld at sympatico.ca Thu Jan 26 03:41:23 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Wed, 25 Jan 2006 21:41:23 -0500 Subject: [Distutils] using bdist_wininst --install-script Message-ID: <43D836D3.4040606@sympatico.ca> Hi there, I'm working on a python application which uses a number of data files. In order to figure out the path to the data files I used to use some hacks, such as letting the application file inject a 'prefix' variable into the root module (package) for other modules to use. This was, IIRC, required since on some systems (notably windows) the actual prefix could differ from any variable set during the packaging stage (since it's the user's choice to override it). Now I read about the '--install-script' option that sounds as if it could solve my problem (on windows). Can anybody comment on this ? How can I use it to fix up 'prefix' variables at installation time ? I'v found the documentation at http://www.python.org/doc/2.4.1/dist/postinstallation-script.html, but that doesn't mention what parameters are available to the script (beside the 'install' / 'remove' flag). Any help is highly appreciated ! Stefan From pje at telecommunity.com Thu Jan 26 04:19:02 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jan 2006 22:19:02 -0500 Subject: [Distutils] using bdist_wininst --install-script In-Reply-To: <43D836D3.4040606@sympatico.ca> Message-ID: <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> At 09:41 PM 1/25/2006 -0500, Stefan Seefeld wrote: >Hi there, > >I'm working on a python application which uses a number of data files. >In order to figure out the path to the data files I used to use some >hacks, such as letting the application file inject a 'prefix' variable >into the root module (package) for other modules to use. The simplest way to do this is to use the package_data option, see: http://docs.python.org/dist/node11.html And then use the __file__ variable of the containing package module to determine the data file locations at runtime. So, if a data file 'bar.dat' is inside your 'foo' package, you can use: import foo pkg_dir = os.path.dirname(foo.__file__) filename = os.path.join(pkg_dir, 'bar.dat') This will work on any platform, with any installation method, bdist_wininst or otherwise. The package_data option was introduced in Python 2.4; if you are using Python 2.3 you can use setuptools instead. There are also hacks out there for even earlier versions of Python that accomplish the same thing, but they're more difficult to implement. Still, even those are far easier than what you're contemplating doing with bdist_wininst. From seefeld at sympatico.ca Thu Jan 26 04:25:30 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Wed, 25 Jan 2006 22:25:30 -0500 Subject: [Distutils] using bdist_wininst --install-script In-Reply-To: <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> References: <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> Message-ID: <43D8412A.1070201@sympatico.ca> Hi Phillip, Phillip J. Eby wrote: > At 09:41 PM 1/25/2006 -0500, Stefan Seefeld wrote: > >> Hi there, >> >> I'm working on a python application which uses a number of data files. >> In order to figure out the path to the data files I used to use some >> hacks, such as letting the application file inject a 'prefix' variable >> into the root module (package) for other modules to use. > > > The simplest way to do this is to use the package_data option, see: > > http://docs.python.org/dist/node11.html > > And then use the __file__ variable of the containing package module to > determine the data file locations at runtime. So, if a data file > 'bar.dat' is inside your 'foo' package, you can use: > > import foo > pkg_dir = os.path.dirname(foo.__file__) > filename = os.path.join(pkg_dir, 'bar.dat') > > This will work on any platform, with any installation method, > bdist_wininst or otherwise. IIUC, this works if and because 'bar.dat' is located somewhere under pkg_dir, right ? The traditional way of installing 'data files' on unix-like systems, however, is to keep them in /share, and the python modules in /lib/python-/site-packages, and there doesn't appear to be a platform-independent path from python module to data dir. Right ? Unfortunately I don't think I can change the directory layout to relocate the data files into a subdirectory of my toplevel python package. Thanks anyways, Stefan From pje at telecommunity.com Thu Jan 26 04:57:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 25 Jan 2006 22:57:39 -0500 Subject: [Distutils] using bdist_wininst --install-script In-Reply-To: <43D8412A.1070201@sympatico.ca> References: <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> At 10:25 PM 1/25/2006 -0500, Stefan Seefeld wrote: >IIUC, this works if and because 'bar.dat' is located somewhere >under pkg_dir, right ? Yes. This is pretty much the only sane way to bundle read-only data files with a Python package. >The traditional way of installing 'data files' >on unix-like systems, however, is to keep them in /share, I believe that's only if they're read-only, in which case there's no reason not to put them in a package directory, tradition notwithstanding. :) >and the python modules in /lib/python-/site-packages, >and there doesn't appear to be a platform-independent path from >python module to data dir. Which is why the sane way to do it for Python is to put them in a package directory; it ensures that you can find them using the same code regardless of platform. >Unfortunately I don't think I can change the directory layout to >relocate the data files into a subdirectory of my toplevel python >package. Why not? From ianb at colorstudy.com Thu Jan 26 06:02:55 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 25 Jan 2006 23:02:55 -0600 Subject: [Distutils] SOURCES not working for me Message-ID: <43D857FF.8040107@colorstudy.com> I'm stumped as to why this package is not working properly: http://svn.pythonpaste.org/RhubarbTart/trunk The package data isn't being included when I build an egg (everything in rhubarbtart/paste_templates/). But SOURCES.txt lists all the proper files. I'm quite baffled. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From pje at telecommunity.com Thu Jan 26 06:39:06 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 00:39:06 -0500 Subject: [Distutils] SOURCES not working for me In-Reply-To: <43D857FF.8040107@colorstudy.com> Message-ID: <5.1.1.6.0.20060126003827.020afa58@mail.telecommunity.com> At 11:02 PM 1/25/2006 -0600, Ian Bicking wrote: >I'm stumped as to why this package is not working properly: > > http://svn.pythonpaste.org/RhubarbTart/trunk > >The package data isn't being included when I build an egg (everything in >rhubarbtart/paste_templates/). But SOURCES.txt lists all the proper >files. I'm quite baffled. The setup.py is missing the 'include_package_data' flag. From strawman at astraw.com Thu Jan 26 09:40:08 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 26 Jan 2006 00:40:08 -0800 Subject: [Distutils] allowing $py_version in ~/.pydistutils.cfg Message-ID: <43D88AE8.7000800@astraw.com> Hi, Attempting to use variable subsitition in my ~/.pydistutils.cfg file (which I'd like to use across both Python 2.3 and Python 2.4), I find that [install] prefix=~/py$py_version_short-$PLAT works fine while the $variables in [easy_install] site_dirs=~/py$py_version_short-$PLAT/lib/python$py_version_short/site-packages don't get expanded. These $variables seem to be an undocumented distutils feature (at least they aren't in the online docs for distutils), but they're nevertheless handy. Would it be possible to include them in setuptools? They are implemented in distutils.util subst_vars(). Cheers! Andrew From pje at telecommunity.com Thu Jan 26 22:23:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 16:23:44 -0500 Subject: [Distutils] [Py2exe-users] Re: py2exe and setuptools w/ install_requires In-Reply-To: <20060120195839.GB4530@kateandchris.net> References: <5.1.1.6.0.20060120123146.0251cd70@mail.telecommunity.com> <20060119232554.GA13618@kateandchris.net> <5.1.1.6.0.20060120123146.0251cd70@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060126161705.020b4310@mail.telecommunity.com> At 02:58 PM 1/20/2006 -0500, Chris Lambacher wrote: >When I build my module eggs, I publish them to a directory >that an http server indexes. I then I have a build script that calls >'easy_install -m -Z -d tmpdir -f http://myserver/eggs module' for each module Note that easy_install now supports using -f with a directory name, so you don't need to have it do an HTTP fetch; just use the local directory path instead of the http: URL. >that I am interested in. Then I add all the new directories in tmpdir to my >python path and call setup.py py2exe. FYI, if you configure the egg directory to be a "site" directory, you won't need to add them to sys.path manually. See: http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installation and replace all the '~/lib/python2.3' references with your desired tmpdir. Then, when you easy_install the eggs to the tmpdir, they will get put on sys.path automatically. >The advantage of this is that each time I build I could potentially be using >a different version of the module. I don't have to worry about not getting >updates to that libarary because I am building a point release of my app on >some strange branch. > >I wanted to make this action triggered by install_requires in the py2exe >command, I am instead doing this externally before the py2exe command. That's going to have to wait until setuptools is more directly integrated with py2exe. At some point, it should be possible for py2exe to just swallow all the needed eggs into the library zipfile, after automatically updating. But that's not going to happen for a while yet. From pje at telecommunity.com Thu Jan 26 22:26:02 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 16:26:02 -0500 Subject: [Distutils] expected non-zero return values when spawning subprocesses In-Reply-To: <43D19138.5020707@sympatico.ca> Message-ID: <5.1.1.6.0.20060126162415.021810a8@mail.telecommunity.com> At 08:41 PM 1/20/2006 -0500, Stefan Seefeld wrote: >Hi there, > >I'm using distutils.spawn.spawn() to run subprocesses during my >build commands. It now happens that one of those subprocesses will >return a non-zero value that doesn't strictly correspond to an error, >and thus I'd like to catch it from within my python code. > >It appears spawn() itself will raise a message containing a stringified >version of the exit code, but not the exit code itself. This seems to >imply that, in order to discriminate on the exit value, I have to >catch DistutilsExecError, and then parse the string. This seems rather >unelegant. Is there some better way ? Could the exception be modified >to contain the original code (as an integral value) ? The answer to both questions is "probably not". The distutils haven't been under active development for some time now. You could perhaps try and submit a patch via the Python Sourceforge patch tracker; perhaps that might get a more authoritative response one way or the other. From pje at telecommunity.com Thu Jan 26 22:29:14 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 16:29:14 -0500 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <614D135D-4D0E-41FA-AFFB-112B90FEEAEB@mac.com> References: <5.1.1.6.0.20060123105955.01e59680@mail.telecommunity.com> <43CF1451.1030600@colorstudy.com> <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <02e401c61b97$73ec3770$bf03030a@trilan> <3f085ecd0601171107md13d0d4td52d00c95d55a70f@mail.gmail.com> <5.1.1.6.0.20060117195446.024c0bb0@mail.telecommunity.com> <43CF1451.1030600@colorstudy.com> <5.1.1.6.0.20060123105955.01e59680@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060126162622.022145d0@mail.telecommunity.com> At 05:39 PM 1/23/2006 +0100, Ronald Oussoren wrote: >On 23-jan-2006, at 17:31, Phillip J. Eby wrote: >>A little experimentation with the socket module shows that I can >>get the >>full list of mirror IPs from Python, so I've changed setuptools in >>SVN to >>just randomly select one to use, which should fix the sticking >>problem on >>Windows (and any other platform where it occurs). >> >>It's not a perfect solution, since of course you can still end up >>with a >>bad mirror for some download, but I'm not sure what else to do, >>short of >>having some kind of option/configuration to control mirror selection. > >You could try several mirrors before giving up. I haven't looked at the >sourcecode for setuptools yet, so I don't know if this is feaseable. It's probably feasible, but a bit of a pain to get set up, and an even bigger pain to test. :) But I've added it to my feature list. From pje at telecommunity.com Thu Jan 26 22:34:51 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 16:34:51 -0500 Subject: [Distutils] Setting distutils options from sitecustomize In-Reply-To: <43D51BEA.4010403@colorstudy.com> Message-ID: <5.1.1.6.0.20060126163013.02188838@mail.telecommunity.com> At 12:09 PM 1/23/2006 -0600, Ian Bicking wrote: >Has anyone tried anything that involves setting the distutils options >(e.g., where to install libraries) from sitecustomize or some other >Python location? I want to put in logic that is more complex than can >be expressed in a configuration file. Setuptools-specific is a-ok too. The standard way to do this with the distutils would be to create custom versions of the commands, or subclass Distribution in order to slap in some extra configuration. For example, you could add to what configuration files get loaded. Of course, that doesn't help you much for affecting *other* people's setup scripts. :) Probably a way to do this in a future version of setuptools would be for me to add a lot more entry points that get run at various phases of a setup script's operation, like when the Distribution loads up configuration files or something. >Jim Fulton asked about something similar on IRC too -- actually about >having an easy_install binary that installed to a different location >than normal (to the Zope instance path). A trivial way to do this currently would be to create a wrapper that changed the current directory to a directory with a custom setup.cfg in it, then ran easy_install. Another way would be to set the HOME environment variable instead, to point to a directory with a ".pydistutils.cfg" in it, although that might lead to other issues. From pje at telecommunity.com Thu Jan 26 22:36:47 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 16:36:47 -0500 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: Message-ID: <5.1.1.6.0.20060126163504.0221cdc8@mail.telecommunity.com> At 01:48 PM 1/23/2006 -0500, Christopher Fonnesbeck wrote: >Is there any way to specify static linking to particular libraries >when building installers with distutils or setuptools? I want to be >able to include the shared libraries that I am linking to in my >builds. I did not see any info on this in the docs. I don't believe there's a way to force linking to occur statically. The distutils does have a 'build_clib' command that can be used to *build* static libraries, which then can be linked with your other code, and the current in-development version of setuptools has limited support for building shared libraries that get included with your package. In both cases, however, the tools are for building the libraries, not including system libraries in your distribution. From pje at telecommunity.com Thu Jan 26 22:45:16 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 16:45:16 -0500 Subject: [Distutils] (extension) module dependencies in setuptools In-Reply-To: <43D5DC3D.7060002@astraw.com> Message-ID: <5.1.1.6.0.20060126163715.03e64118@mail.telecommunity.com> At 11:50 PM 1/23/2006 -0800, Andrew Straw wrote: >I would like to propose a feature for setuptools: runtime enforcement of >dependencies specified at build time by setup.py. I appreciate that >"pkg_resources.require('foo==1.0')" works, but this requires a tedious >update of version numbers in affected source files every time you >upgrade foo and rebuild the target package. I'm thinking, in particular, >of extension modules built on a particular version of another package >with its own C interface. Think matplotlib.backends._ns_backend_agg >depending on numpy. I'm getting close to starting on an overhaul of setuptools' experimental shared library support to allow dynamic linking to libraries included in a different egg than the one the extension is in. When I do that, there's definitely going to be some inter-egg dependency checking of sorts, except it will be based on *exact* version numbers, because it will be tied to the egg version you linked against. I suppose it's possible I could do something similar for other sorts of extensions, but it seems to me that the simple way to do inter-extension API checks in Python today is to have a .h supplied by one extension containing a version, and having a PyCObject you import from the target extension with a function pointer that gets the built version. You then compare the API version you compiled against with the API version present at runtime. The source code doesn't change; you're just verifying that the API version matches what you compiled against. Notice that this doesn't require setuptools in order to be useful. It's simply part of best practices for C extensions that provide a C API for other extensions to use. Python itself uses this technique as well. >I'm already attempting to dispel enough anti-egg sentiment (for reasons >I don't understand) resulting from slight changes to setup.py. Cognitive dissonance is sometimes a harsh taskmaster. :) Luckily my suggestion is entirely independent of eggs and setuptools. From robert.kern at gmail.com Thu Jan 26 22:46:11 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 26 Jan 2006 15:46:11 -0600 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: References: Message-ID: Christopher Fonnesbeck wrote: > Is there any way to specify static linking to particular libraries when > building installers with distutils or setuptools? I want to be able to > include the shared libraries that I am linking to in my builds. I did > not see any info on this in the docs. Not really, no. In many cases (e.g., GNU ld), there's simply no way to tell the linker that you prefer static libraries to shared libraries when you are building a shared library like a Python extension. You simply have to make sure that the static libraries are found first. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pje at telecommunity.com Thu Jan 26 22:56:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 16:56:57 -0500 Subject: [Distutils] allowing $py_version in ~/.pydistutils.cfg In-Reply-To: <43D88AE8.7000800@astraw.com> Message-ID: <5.1.1.6.0.20060126164941.02185a28@mail.telecommunity.com> At 12:40 AM 1/26/2006 -0800, Andrew Straw wrote: >Hi, > >Attempting to use variable subsitition in my ~/.pydistutils.cfg file >(which I'd like to use across both Python 2.3 and Python 2.4), I find that > >[install] >prefix=~/py$py_version_short-$PLAT > >works fine > >while the $variables in > >[easy_install] >site_dirs=~/py$py_version_short-$PLAT/lib/python$py_version_short/site-packages > >don't get expanded. > >These $variables seem to be an undocumented distutils feature (at least >they aren't in the online docs for distutils), but they're nevertheless >handy. Would it be possible to include them in setuptools? They are >implemented in distutils.util subst_vars(). Actually they're implemented by distutils.command.install.install, but it looks fairly straightforward to add. I've put it on my to-do list. From smurf at smurf.noris.de Thu Jan 26 23:02:12 2006 From: smurf at smurf.noris.de (Matthias Urlichs) Date: Thu, 26 Jan 2006 23:02:12 +0100 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: References: Message-ID: <20060126220212.GA7999@kiste.smurf.noris.de> Hi, Robert Kern: > Not really, no. In many cases (e.g., GNU ld), there's simply no way to > tell the linker that you prefer static libraries to shared libraries > when you are building a shared library like a Python extension. You > simply have to make sure that the static libraries are found first. > Note: Linking static code (gcc without -fpic => foo.o => libfoo.a) into a shared library (lib*.so) is a bad idea. On i386, the required relocation means that the static library's text is not shared between concurrent invocations, and loading the library is more expensive too. On some other architectures, either linking the library, or loading it, will simply fail. You need to build an archive of shared objects (same as above, but use -fpic or equivalent). These are usually named lib*_pic.a IIRC. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf at smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - It is surely a great calamity for a human being to have no obsessions. -- Robert Bly -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060126/a37ba568/attachment-0001.pgp From seefeld at sympatico.ca Thu Jan 26 23:05:50 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Thu, 26 Jan 2006 17:05:50 -0500 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: References: Message-ID: <43D947BE.8080003@sympatico.ca> Robert Kern wrote: > Christopher Fonnesbeck wrote: > >>Is there any way to specify static linking to particular libraries when >>building installers with distutils or setuptools? I want to be able to >>include the shared libraries that I am linking to in my builds. I did >>not see any info on this in the docs. > > > Not really, no. In many cases (e.g., GNU ld), there's simply no way to tell the > linker that you prefer static libraries to shared libraries when you are > building a shared library like a Python extension. You simply have to make sure > that the static libraries are found first. I believe this is wrong. The (GNU ld) man pages mention '-Bdynamic' and '-Bstatic' as a means to instruct the linker to link to a particular library dynamically or statically. (To only link statically to libbar.a you could write e.g. ... -lfoo -Bstatic -lbar -Bdynamic -lbaz etc.) However, the way distutils is written makes it practically impossible to control such behavior without completely rewriting the whole 'build_ext' command. Hopefully setuptools can manage this better. Regards, Stefan From robert.kern at gmail.com Thu Jan 26 23:12:29 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 26 Jan 2006 16:12:29 -0600 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: <20060126220212.GA7999@kiste.smurf.noris.de> References: <20060126220212.GA7999@kiste.smurf.noris.de> Message-ID: Matthias Urlichs wrote: > Hi, > > Robert Kern: > >>Not really, no. In many cases (e.g., GNU ld), there's simply no way to >>tell the linker that you prefer static libraries to shared libraries >>when you are building a shared library like a Python extension. You >>simply have to make sure that the static libraries are found first. > > Note: Linking static code (gcc without -fpic => foo.o => libfoo.a) into > a shared library (lib*.so) is a bad idea. On i386, the required > relocation means that the static library's text is not shared between > concurrent invocations, and loading the library is more expensive too. > On some other architectures, either linking the library, or loading it, > will simply fail. > > You need to build an archive of shared objects (same as above, but use > -fpic or equivalent). These are usually named lib*_pic.a IIRC. On OS X (the platform Chris is building for), -fPIC is the default for everything, so the operation is fairly safe, if not optimal. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From strawman at astraw.com Thu Jan 26 23:16:17 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 26 Jan 2006 14:16:17 -0800 Subject: [Distutils] (extension) module dependencies in setuptools In-Reply-To: <5.1.1.6.0.20060126163715.03e64118@mail.telecommunity.com> References: <5.1.1.6.0.20060126163715.03e64118@mail.telecommunity.com> Message-ID: <43D94A31.8080405@astraw.com> Phillip J. Eby wrote: > At 11:50 PM 1/23/2006 -0800, Andrew Straw wrote: > >> I would like to propose a feature for setuptools: runtime enforcement of >> dependencies specified at build time by setup.py. I appreciate that >> "pkg_resources.require('foo==1.0')" works, but this requires a tedious >> update of version numbers in affected source files every time you >> upgrade foo and rebuild the target package. I'm thinking, in particular, >> of extension modules built on a particular version of another package >> with its own C interface. Think matplotlib.backends._ns_backend_agg >> depending on numpy. > > > I'm getting close to starting on an overhaul of setuptools' > experimental shared library support to allow dynamic linking to > libraries included in a different egg than the one the extension is > in. When I do that, there's definitely going to be some inter-egg > dependency checking of sorts, except it will be based on *exact* > version numbers, because it will be tied to the egg version you linked > against. > > I suppose it's possible I could do something similar for other sorts > of extensions, but it seems to me that the simple way to do > inter-extension API checks in Python today is to have a .h supplied by > one extension containing a version, and having a PyCObject you import > from the target extension with a function pointer that gets the built > version. You then compare the API version you compiled against with > the API version present at runtime. The source code doesn't change; > you're just verifying that the API version matches what you compiled > against. > > Notice that this doesn't require setuptools in order to be useful. > It's simply part of best practices for C extensions that provide a C > API for other extensions to use. Python itself uses this technique as > well. Absolutely. In fact, I just implemented exactly this in numpy. I was thinking, however, of the case of older extension modules which don't do this. I suppose I could submit patches there, too, but there are only so many hours in the day... Anyhow, when you implement version checking for shared libraries, if it looks straightforward to implement it for other extensions, you have my vote for going ahead and adding it in... Cheers! Andrew From pje at telecommunity.com Thu Jan 26 23:59:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 17:59:11 -0500 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: <43D947BE.8080003@sympatico.ca> References: Message-ID: <5.1.1.6.0.20060126175601.02216130@mail.telecommunity.com> At 05:05 PM 1/26/2006 -0500, Stefan Seefeld wrote: >I believe this is wrong. The (GNU ld) man pages mention '-Bdynamic' and >'-Bstatic' >as a means to instruct the linker to link to a particular library >dynamically or >statically. (To only link statically to libbar.a you could write e.g. > >... -lfoo -Bstatic -lbar -Bdynamic -lbaz > >etc.) > >However, the way distutils is written makes it practically impossible to >control >such behavior without completely rewriting the whole 'build_ext' command. >Hopefully setuptools can manage this better. Probably not, as I'm not rewriting build_ext, just wrapping it. You might try kludging your library names to be 'foo -Bstatic', 'bar -Bdynamic', etc., if you're on the right platform/compiler. From robert.kern at gmail.com Fri Jan 27 00:29:35 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 26 Jan 2006 17:29:35 -0600 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: <43D947BE.8080003@sympatico.ca> References: <43D947BE.8080003@sympatico.ca> Message-ID: Stefan Seefeld wrote: > Robert Kern wrote: >>Not really, no. In many cases (e.g., GNU ld), there's simply no way to tell the >>linker that you prefer static libraries to shared libraries when you are >>building a shared library like a Python extension. You simply have to make sure >>that the static libraries are found first. > > I believe this is wrong. The (GNU ld) man pages mention '-Bdynamic' and '-Bstatic' > as a means to instruct the linker to link to a particular library dynamically or > statically. (To only link statically to libbar.a you could write e.g. > > ... -lfoo -Bstatic -lbar -Bdynamic -lbaz > > etc.) Ah, yes, you are correct. Unfortunately, OS X's ld does not have these options (and is not GNU; I am an idiot). -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From bob at redivi.com Fri Jan 27 01:02:44 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 26 Jan 2006 16:02:44 -0800 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: References: <43D947BE.8080003@sympatico.ca> Message-ID: On Jan 26, 2006, at 3:29 PM, Robert Kern wrote: > Stefan Seefeld wrote: >> Robert Kern wrote: > >>> Not really, no. In many cases (e.g., GNU ld), there's simply no >>> way to tell the >>> linker that you prefer static libraries to shared libraries when >>> you are >>> building a shared library like a Python extension. You simply >>> have to make sure >>> that the static libraries are found first. >> >> I believe this is wrong. The (GNU ld) man pages mention '- >> Bdynamic' and '-Bstatic' >> as a means to instruct the linker to link to a particular library >> dynamically or >> statically. (To only link statically to libbar.a you could write e.g. >> >> ... -lfoo -Bstatic -lbar -Bdynamic -lbaz >> >> etc.) > > Ah, yes, you are correct. Unfortunately, OS X's ld does not have > these options > (and is not GNU; I am an idiot). What I normally do is specify the full path to the library in extra_link_args, and remove it from the libraries list. -bob From chris at kateandchris.net Fri Jan 27 01:08:45 2006 From: chris at kateandchris.net (Chris Lambacher) Date: Thu, 26 Jan 2006 19:08:45 -0500 Subject: [Distutils] [Py2exe-users] Re: py2exe and setuptools w/ install_requires In-Reply-To: <5.1.1.6.0.20060126161705.020b4310@mail.telecommunity.com> References: <5.1.1.6.0.20060120123146.0251cd70@mail.telecommunity.com> <20060119232554.GA13618@kateandchris.net> <5.1.1.6.0.20060120123146.0251cd70@mail.telecommunity.com> <5.1.1.6.0.20060126161705.020b4310@mail.telecommunity.com> Message-ID: <20060127000845.GA13449@kateandchris.net> On Thu, Jan 26, 2006 at 04:23:44PM -0500, Phillip J. Eby wrote: > At 02:58 PM 1/20/2006 -0500, Chris Lambacher wrote: > >When I build my module eggs, I publish them to a directory > >that an http server indexes. I then I have a build script that calls > >'easy_install -m -Z -d tmpdir -f http://myserver/eggs module' for each > >module > > Note that easy_install now supports using -f with a directory name, so you > don't need to have it do an HTTP fetch; just use the local directory path > instead of the http: URL. I tried to use this and it then did not seem to honor -d tmpdir, it just tried to add the egg in place to the path. > > > >that I am interested in. Then I add all the new directories in tmpdir to > >my > >python path and call setup.py py2exe. > > FYI, if you configure the egg directory to be a "site" directory, you won't > need to add them to sys.path manually. See: > > http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installation > > and replace all the '~/lib/python2.3' references with your desired > tmpdir. Then, when you easy_install the eggs to the tmpdir, they will get > put on sys.path automatically. I specifically don't want to do this because I don't actually want the package installed, I just want it in the path for the duration of the call to setup.py py2exe Note that I can't just install it and then uninstall it because there could be more than one build happening at the same time, each wanting a different version of the item being installed. > > > >The advantage of this is that each time I build I could potentially be > >using > >a different version of the module. I don't have to worry about not getting > >updates to that libarary because I am building a point release of my app on > >some strange branch. > > > >I wanted to make this action triggered by install_requires in the py2exe > >command, I am instead doing this externally before the py2exe command. > > That's going to have to wait until setuptools is more directly integrated > with py2exe. At some point, it should be possible for py2exe to just > swallow all the needed eggs into the library zipfile, after automatically > updating. But that's not going to happen for a while yet. > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Py2exe-users mailing list > Py2exe-users at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/py2exe-users From ianb at colorstudy.com Fri Jan 27 03:34:44 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 26 Jan 2006 20:34:44 -0600 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <5.1.1.6.0.20060117194917.00a7c7b0@mail.telecommunity.com> References: <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <5.1.1.6.0.20060117194917.00a7c7b0@mail.telecommunity.com> Message-ID: <43D986C4.4010805@colorstudy.com> Phillip J. Eby wrote: >>>Hi all. Ian Bicking reported an issue with Sourceforge's download process >>>changing again; I've updated setuptools in SVN but haven't made a new >>>release yet. If you need the fixed version, update to the development >>>version via "ez_setup.py setuptools==dev" for now. Thanks. >> >>This is a probably a FAQ, but maybe the Sourceforge guys can be talked into >>providing some machine-friendly way for downloading files? > > > They have already done it, and I've changed setuptools to use it now (as of > r42088). I've now removed all of the code originally contributed by Ian to > parse SourceForge's "human user" HTML and simply preprocess > prdownloads.sf.net URLs to a similar path on dl.sourceforge.net, which is a > round-robin DNS for the various mirrors. This cuts two page fetches out of > the process and eliminates any HTML scraping dependencies. As long as SF > doesn't get rid of the *.dl system, this should continue to work no matter > how much the HTML frontend changes. So far my experiences downloading this way have been very poor -- it fails more often than not, often stalling indefinitely. What are other people's experiences? If my experience is typical, I think it would be better to go back to the regex way with a fixed regex -- it may be fragile in the long term, but it works much better for the moment. To deal with this for Paste, I even asked Tavis to put a Cheetah package on the Cheese Shop page, but since the SF link is longer easy_install prefers it :( -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From seefeld at sympatico.ca Fri Jan 27 04:29:11 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Thu, 26 Jan 2006 22:29:11 -0500 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: <5.1.1.6.0.20060126175601.02216130@mail.telecommunity.com> References: <5.1.1.6.0.20060126175601.02216130@mail.telecommunity.com> Message-ID: <43D99387.5080509@sympatico.ca> Phillip J. Eby wrote: >> Hopefully setuptools can manage this better. > > > Probably not, as I'm not rewriting build_ext, just wrapping it. Is this wrapping transparent ? Or do you present your own command interfaces, for which the current implementation is simply distutils' commands ? One of the weak aspects of distutils (IMO) is its lack of extensibility. There is just no way to plug my own 'Extension' in (with all its build infrastructure), for example. This is definitely a lesson to be learned. FWIW, Stefan From bob at redivi.com Fri Jan 27 04:47:25 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 26 Jan 2006 19:47:25 -0800 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: <43D99387.5080509@sympatico.ca> References: <5.1.1.6.0.20060126175601.02216130@mail.telecommunity.com> <43D99387.5080509@sympatico.ca> Message-ID: <48A90EB7-AD5C-4540-B2CA-B81D3CB1C939@redivi.com> On Jan 26, 2006, at 7:29 PM, Stefan Seefeld wrote: > Phillip J. Eby wrote: > >>> Hopefully setuptools can manage this better. >> >> >> Probably not, as I'm not rewriting build_ext, just wrapping it. > > Is this wrapping transparent ? Or do you present your own command > interfaces, for which the current implementation is simply distutils' > commands ? > One of the weak aspects of distutils (IMO) is its lack of > extensibility. > There is just no way to plug my own 'Extension' in (with all its build > infrastructure), for example. > This is definitely a lesson to be learned. While I don't agree with a lot of the design decisions in distutils, it's plenty extensible. You can plug your own Extension in, just subclass it. -bob From pje at telecommunity.com Fri Jan 27 05:07:31 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Jan 2006 23:07:31 -0500 Subject: [Distutils] Sourceforge changed HTML again... In-Reply-To: <43D986C4.4010805@colorstudy.com> References: <5.1.1.6.0.20060117194917.00a7c7b0@mail.telecommunity.com> <5.1.1.6.0.20060117143758.00a68d70@mail.telecommunity.com> <5.1.1.6.0.20060117194917.00a7c7b0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060126230434.02008ae0@mail.telecommunity.com> At 08:34 PM 1/26/2006 -0600, Ian Bicking wrote: >So far my experiences downloading this way have been very poor -- it fails >more often than not, often stalling indefinitely. What are other people's >experiences? If my experience is typical, I think it would be better to >go back to the regex way with a fixed regex -- it may be fragile in the >long term, but it works much better for the moment. What platform are you using, and what revision of easy_install? The first version of the dl.sf.net trick could end up using the same IP over and over depending on local DNS cache settings. Windows machines were particularly prone to getting stuck with a bad IP. The current version explicitly requests *all* of the IPs, then selects from them at random. >To deal with this for Paste, I even asked Tavis to put a Cheetah package >on the Cheese Shop page, but since the SF link is longer easy_install >prefers it :( There's probably a different reason why. All else being equal, easy_install prefers *shorter* URLs for the same target version of a package. From seefeld at sympatico.ca Fri Jan 27 05:10:26 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Thu, 26 Jan 2006 23:10:26 -0500 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: <48A90EB7-AD5C-4540-B2CA-B81D3CB1C939@redivi.com> References: <5.1.1.6.0.20060126175601.02216130@mail.telecommunity.com> <43D99387.5080509@sympatico.ca> <48A90EB7-AD5C-4540-B2CA-B81D3CB1C939@redivi.com> Message-ID: <43D99D32.8050609@sympatico.ca> Bob Ippolito wrote: > While I don't agree with a lot of the design decisions in distutils, > it's plenty extensible. You can plug your own Extension in, just > subclass it. I disagree. An 'Extension' is "Just a collection of attributes that describes an extension module and everything needed to build it..." (to cite the Extension's own doc string). There is no encapsulation of behavior (i.e. how to build it), as that is handled in separate command classes. And this is why I provided my own build_ext class, which, however, didn't use anything from the 'Extension' type, since that isn't adequate for my needs. Regards, Stefan From trentm at ActiveState.com Fri Jan 27 09:33:51 2006 From: trentm at ActiveState.com (Trent Mick) Date: Fri, 27 Jan 2006 00:33:51 -0800 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: References: <43D947BE.8080003@sympatico.ca> Message-ID: <20060127083351.GB27526@activestate.com> [Bob Ippolito wrote] > > On Jan 26, 2006, at 3:29 PM, Robert Kern wrote: > > > Stefan Seefeld wrote: > >> Robert Kern wrote: > > > >>> Not really, no. In many cases (e.g., GNU ld), there's simply no > >>> way to tell the > >>> linker that you prefer static libraries to shared libraries when > >>> you are > >>> building a shared library like a Python extension. You simply > >>> have to make sure > >>> that the static libraries are found first. > >> > >> I believe this is wrong. The (GNU ld) man pages mention '- > >> Bdynamic' and '-Bstatic' > >> as a means to instruct the linker to link to a particular library > >> dynamically or > >> statically. (To only link statically to libbar.a you could write e.g. > >> > >> ... -lfoo -Bstatic -lbar -Bdynamic -lbaz > >> > >> etc.) > > > > Ah, yes, you are correct. Unfortunately, OS X's ld does not have > > these options > > (and is not GNU; I am an idiot). > > What I normally do is specify the full path to the library in > extra_link_args, and remove it from the libraries list. Ditto. Trent -- Trent Mick TrentM at ActiveState.com From david at mantara.com Fri Jan 27 09:42:02 2006 From: david at mantara.com (David Arnold) Date: Fri, 27 Jan 2006 19:42:02 +1100 Subject: [Distutils] static linking in distutils/setuptools? In-Reply-To: Your message of "Thu, 26 Jan 2006 15:46:11 MDT." Message-ID: -->"Robert" == Robert Kern writes: Robert> Not really, no. In many cases (e.g., GNU ld), there's simply Robert> no way to tell the linker that you prefer static libraries to Robert> shared libraries when you are building a shared library like a Robert> Python extension. You simply have to make sure that the static Robert> libraries are found first. you can pass the linker a direct path to the archive, rather than using -L/-l. this will ensure static linkage. (not sure if that's helpful in this context though) d From seefeld at sympatico.ca Fri Jan 27 17:25:58 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Fri, 27 Jan 2006 11:25:58 -0500 Subject: [Distutils] using bdist_wininst --install-script In-Reply-To: <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> References: <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> Message-ID: <43DA4996.7010103@sympatico.ca> Phillip J. Eby wrote: >> Unfortunately I don't think I can change the directory layout to >> relocate the data files into a subdirectory of my toplevel python >> package. > > > Why not? because my python modules are part of a multi-language package (to be more specific: a framework with multiple language bindings), so a number of conventions are to be respected (such as http://www.debian.org/doc/packaging-manuals/fhs/). Regards, Stefan From pje at telecommunity.com Fri Jan 27 18:32:52 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 27 Jan 2006 12:32:52 -0500 Subject: [Distutils] using bdist_wininst --install-script In-Reply-To: <43DA4996.7010103@sympatico.ca> References: <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060127121851.0202de80@mail.telecommunity.com> At 11:25 AM 1/27/2006 -0500, Stefan Seefeld wrote: >Phillip J. Eby wrote: > > >> Unfortunately I don't think I can change the directory layout to > >> relocate the data files into a subdirectory of my toplevel python > >> package. > > > > > > Why not? > >because my python modules are part of a multi-language package >(to be more specific: a framework with multiple language bindings), >so a number of conventions are to be respected (such as >http://www.debian.org/doc/packaging-manuals/fhs/). """If an application uses a subdirectory, all architecture-dependent data exclusively used by the application should be placed within that subdirectory. For example, the perl5 subdirectory for Perl 5 modules and libraries.""" Sounds to me like there are only two ways to interpret this statement. One leads to the conclusion that Perl or Python modules must *not* live under /usr/lib (since they are *not* architecture-dependent), or that it's fine to place data files alongside modules. Since the former conclusion is not the practice, I interpret this to mean that the latter applies. A simple thought experiment should suffice to make the point more clear: if you took the data files and embedded them inside a Python module using a string and a base64 encoding, would the module then have to live in /usr/share? What if a C library includes static data in the library? Does that make it belong in /usr/share? What about Java jars or libraries containing resource files? Do those go in /usr/share? Since Python modules are often installed with source code, doesn't that mean they belong in /usr/src, with only the compiled libraries in /usr/lib? As far as I can tell, the FHS is already interpreted quite flexibly enough to easily allow for embedded data files in Python packages. Anybody who uses that as an excuse to denigrate your package either lacks imagination or is using the FHS as a political tool rather than a technical guideline. :) From seefeld at sympatico.ca Fri Jan 27 23:00:26 2006 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Fri, 27 Jan 2006 17:00:26 -0500 Subject: [Distutils] using bdist_wininst --install-script In-Reply-To: <5.1.1.6.0.20060127121851.0202de80@mail.telecommunity.com> References: <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> <5.1.1.6.0.20060127121851.0202de80@mail.telecommunity.com> Message-ID: <43DA97FA.4030300@sympatico.ca> Phillip J. Eby wrote: > At 11:25 AM 1/27/2006 -0500, Stefan Seefeld wrote: > >> Phillip J. Eby wrote: >> >> >> Unfortunately I don't think I can change the directory layout to >> >> relocate the data files into a subdirectory of my toplevel python >> >> package. >> > >> > >> > Why not? >> >> because my python modules are part of a multi-language package >> (to be more specific: a framework with multiple language bindings), >> so a number of conventions are to be respected (such as >> http://www.debian.org/doc/packaging-manuals/fhs/). > > > """If an application uses a subdirectory, all architecture-dependent > data exclusively used by the application should be placed within that > subdirectory. For example, the perl5 subdirectory for Perl 5 modules and > libraries.""" > > Sounds to me like there are only two ways to interpret this statement. > One leads to the conclusion that Perl or Python modules must *not* live > under /usr/lib (since they are *not* architecture-dependent), or that > it's fine to place data files alongside modules. Since the former > conclusion is not the practice, I interpret this to mean that the latter > applies. Sorry, may be I didn't express myself very clearly; I'll try again. I have a number of 'architecture-independent' files that are used by different parts of the framework, such as python modules, C++ libraries and applets, etc. I can't see any good reason why my C++ code (which doesn't even know that there are python modules as part of the same framework) should look for its data in some obscure 'site-package' directory. :-) The canonical way for such data files is still /share. Regards, Stefan From pje at telecommunity.com Fri Jan 27 23:04:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 27 Jan 2006 17:04:10 -0500 Subject: [Distutils] using bdist_wininst --install-script In-Reply-To: <43DA97FA.4030300@sympatico.ca> References: <5.1.1.6.0.20060127121851.0202de80@mail.telecommunity.com> <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125221443.020add40@mail.telecommunity.com> <5.1.1.6.0.20060125225447.020aeb40@mail.telecommunity.com> <5.1.1.6.0.20060127121851.0202de80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060127170248.02026e70@mail.telecommunity.com> At 05:00 PM 1/27/2006 -0500, Stefan Seefeld wrote: >Sorry, may be I didn't express myself very clearly; I'll try again. > >I have a number of 'architecture-independent' files that are used by >different parts of the framework, such as python modules, C++ libraries >and applets, etc. >I can't see any good reason why my C++ code (which doesn't even know >that there are python modules as part of the same framework) should look >for its data in some obscure 'site-package' directory. :-) Ah. :) Maybe the best solution then would be to have a way to generate C++ and Python (and whatever other language) modules with the data embedded in them, then, so that none of them have to worry about locations. :)