Mailman 3 October 2006 - Distutils-SIG

setuptools site-package pollution
by Anthony Tarlano Oct. 10, 2006

Oct. 10, 2006

Hi all.. Firstly, I would like to say thanks for the setuptools package, which I was introduced to after reading about the RuleDispatch package on the IBM developerworks charming python series. Oh btw. RuleDispatch is the most useful python package that I have seen in the last year in the python world, so thanks for that also ;) Now that the congrats and hugs are out of the way, I would like to ask a question. How can I tell setuptools not to put packages, such as dispatch, protocols, and setuptools, in a subdirectory named the same as the egg and to just put the package name. For example from pydoc I currently get: c:\python\lib\site-packages\ruledispatch-0.5a0.dev_r2100-py2.4-win32.egg dispatch (package) c:\python\lib\site-packages\pyprotocols-1.0a0dev_r2082-py2.4-win32.egg dispatch protocols (package) c:\python\lib\site-packages\setuptools-0.6a8-py2.4.egg easy_install pkg_resources setuptools (package) site what I would like to see is just: ..\site-packages dispatch (package) ..\site-packages protocols (package) ..\site-packages setuptools (package) Maybe it is just a pet-peeve, but I like to keep a nice tidy site-packages directory, and these long directory names just seem to me to be pollution of my site-packages directory in my command shell. If there is an option to have just the packages, I would really appreciate someone telling me what it is. If not maybe it could be considered, and to put whatever meta-data the directory names are providing somewhere else.. Thanks, Anthony

3 2

compiler selection with bdist_egg of setuptools
by Venkat Bommakanti Oct. 9, 2006

Oct. 9, 2006

Hi, While compiling some py/C source pkgs, such as: . RuleDispatch-0.5a0.dev-r2115.tar.gz . cElementTree-1.0.5-20051216.zip . PyProtocols-1.0a0dev_r2082.zip . Cheetah-2.0rc7.tar.gz I noticed in the logs that each (after being unpacked) has the following command invoked: setup.py -q bdist_egg --dist-dir /tmp/.... Since selecting a 'compiler' is not an option for the "bdist_egg" command, are even the settings in ~/.pydistutils.cfg file: [build] compiler = unix [bdist_egg] plat_name = ppc picked up ? They appear not to be picked up, and I'm suspecting that since the [build] settings are valid only for the "build" command, it will not have any effect on the "bdist_egg" command used by all the above comps. Is that accurate ? If plat_name is not picked up, how is setuptools figuring out that the platform is i686 (given that the build is being done for a i686 linux box). Verbose logs don't show where/how that decision is made... Thanks, /venkat

2 1

Re: [Distutils] cross compiling using setuptools
by Venkat Bommakanti Oct. 9, 2006

Oct. 9, 2006

>>Based on this suggestion, for the cross compile case, I tried to set: >> [bdist_egg] >> plat_name = ppc >>but, it still insists on falling back to i686 for plat_name. Could be I'm >>doing something fundamentally wrong. >Dunno. Is this in $HOME/.pydistutils.cfg? For the cross compile case (i686 build box -> ppc target), rather than the setup.py/setup.cfg option in the TurboGears src/pkg dir, I thought I'll try the basic method of using $HOME/.pydistutils.cfg first. So I created one ($HOME/.pydistutils.cfg) with the following contents: [build] compiler=powerpc-unknown-linux-gcc [bdist_egg] pla_name=ppc When I run: easy_install --prefix=usr -s usr/bin -f . TG.egg I get an error msg: error: Setup script exited with error: don't know how to compile C/C++ code on platform 'posix' with 'powerpc-unknown-linux-gcc' compiler I have 'powerpc-unknown-linux-gcc' in my path. Could it be that its not picking up some include dirs etc. I also tried the same with i686 build box -> i686 target, pointing to my i686 gcc 'i686-pc-linux-gcc'... [build] compiler=i686-pc-linux-gcc [bdist_egg] pla_name=i686 I get the same error... error: Setup script exited with error: don't know how to compile C/C++ code on platform 'posix' with 'i686-pc-linux-gcc' compiler Any idea what may be wrong here... Thanks much, /venkat

2 1

Re: [Distutils] cross compiling using setuptools
by Venkat Bommakanti Oct. 9, 2006

Oct. 9, 2006

>>Based on this suggestion, for the cross compile case, I tried to set: >> [bdist_egg] >> plat_name = ppc >>but, it still insists on falling back to i686 for plat_name. Could be I'm >>doing something fundamentally wrong. >Dunno. Is this in $HOME/.pydistutils.cfg? I'm actually using a seperate <temp_dir> for each <compile_target>. So, they are: <top-of-src-tree>/temp_i686/setup.cfg and <top-of-src-tree>/temp_ppc/setup.cfg They have their own (temp) set of TG comps in their native tar/zip formats. This will allow for a developer to snap a tree and build from there, without any build-specifc files in their $HOMEs. Each will have <top-of-src-tree>/temp_<tgt>/easy_install -f . TurboGears invoked from their respective directories, and will use their respective setup.cfg files. >> 'am also >>assuming the syntax is the same for setup.cfg (local to TG source dir) and >>pydistutils.cfg ($HOME). >On Unix-like platforms, it's ".pydistutils.cfg" (note the '.' at the >beginning), NOT "pydistutils.cfg". The latter name is used only on Windows. Based on the above layout, and from what I've understood so far, I'm assuming: <top-of-src-tree>/temp_<tgt>/setup.cfg is the equivalent of: $HOME/.pydistutils.cfg per the responses so far, and notes in pages like: http://docs.python.org/inst/config-syntax.html Is is accurate to say that the contents of setup.cfg & pydistutils.cfg are the exactly the same ? Thanks a lot, /venkat

2 1

Re: [Distutils] cross compiling using setuptools
by Venkat Bommakanti Oct. 9, 2006

Oct. 9, 2006

>>Assuming I can use setuptools as-is and "easy_install ..." invoke syntax >>like above - for the cross build (ppc target), how would I specify the >>options meant for: >> . setup.py (of each of the above comps): >> . -c (cross-gcc) >> . bdest_egg (setuptools): >> . --dist-dir >> . --plat-name >> . exclude-source >>in the "easy_install ..." command. I read: >Put them in your ~/.pydistutils.cfg file, using e.g. >[build] >compiler = ... >[bdist_egg] >plat_name = ... >exclude_source = 1 Based on this suggestion, for the cross compile case, I tried to set: [bdist_egg] plat_name = ppc but, it still insists on falling back to i686 for plat_name. Could be I'm doing something fundamentally wrong. 'am also trying to "set" as many params to easy_install / setup.py-of-comp (in TG's suite of components) as possible this way... but its unclear if I'm doing them right... Is there a reference somewhere in (http://peak.telecommunity.com/) to all . the sections that can go into these files, and . the key/value pairs of settings, that can go into each section. For example, I couldn't see anywhere it documented that [bdist_egg] is one of the many valid sections in these files. I'm not doubting the earlier statement, but just unable to find a single place that lists all the possible cases. 'am also assuming the syntax is the same for setup.cfg (local to TG source dir) and pydistutils.cfg ($HOME). Thanks, /venkat

2 1

Re: [Distutils] cross compiling using setuptools
by Venkat Bommakanti Oct. 8, 2006

Oct. 8, 2006

>At 02:24 PM 10/8/2006 -0700, Venkat Bommakanti wrote: >> Wondering if a "setup.cfg" for this component (TG) in the component's >> directory itself, will serve the purpose. >Well, if each project is going to be built for only one platform, >sure. But the setup.cfg *overrides* the .pydistutils.cfg, so you don't >want to distribute a bunch of site/platform-specific options in the >project's published setup.cfg. >In any case, I thought your original issue had to do with getting >easy_install to handle these packages automatically, without modifying the >original packages in any way. And that's the context in which I've been >answering your questions. I apologize for the confusion here... Will try to clarify the full need... Basically, the need is such: . a developer would snap a <src> tree . build python <target>, TG for a <target> of choice (i686/ppc) A developer could potentially fire the build of both targets simultaneously. And to build the ppc python/TG (cross target) they'd use the necessary i686 python that is part of a <tools> directory on the i686 build box. Under those circumstances, and also the desire to keep all build related files (incuding .pydistutils.cfg or equivalents) within the source tree itself, I wanted to know the best recourse. Using $HOME/.pydistutils.cfg may not be a viable option due the fact that it'd reside outside the <snapped> source tree, and could become a maintenance issue... Would like a solution where one gets the cfg file (or equivalent) when they snap a tree direct... I was hoping something like: <top-of-src-tree>/TurboGears/<pydistutils.cfg or setup.cfg equivalent> with info relevant to: [build] compiler = $COMPILER (native or cross-gcc) [bdist_egg] plat_name = $TARGET (i686 or ppc) exclude_source = 1 would let me specify (target-based) info that is picked up according to the build target. Thanks again, /venkat

2 1

Re: [Distutils] cross compiling using setuptools
by Venkat Bommakanti Oct. 8, 2006

Oct. 8, 2006

>At 02:07 PM 10/8/2006 -0700, Venkat Bommakanti wrote: >>So, how would I specify that variance in just **one** ~/.pydistutils.cfg >>OR is there a way to have setuptools pickup different versions of this >>file, based on the <target> chosen for the makefile. >You could use different values of $HOME, since that's where the per-user >config file is read from. Wondering if a "setup.cfg" for this component (TG) in the component's directory itself, will serve the purpose. That ways, each developer wouldn't have maintain thier own ~/.pydistutils.cfg files. Thanks again, /venkat

2 1

Re: [Distutils] cross compiling using setuptools
by Venkat Bommakanti Oct. 8, 2006

Oct. 8, 2006

Thanks Phillip. Have a followup question below. /venkat >>Assuming I can use setuptools as-is and "easy_install ..." invoke syntax >>like above - for the cross build (ppc target), how would I specify the >>options meant for: >> . setup.py (of each of the above comps): >> >> . -c (cross-gcc) >> . bdest_egg (setuptools): >> . --dist-dir >> . --plat-name >> . exclude-source >>in the "easy_install ..." command. I read: >> >>http://peak.telecommunity.com/DevCenter/setuptools#bdist-egg-create-a-pytho… >>but its unclear how the above options are passed on to "easy_install ...". >Put them in your ~/.pydistutils.cfg file, using e.g. >[build] >compiler = ... >[bdist_egg] >plat_name = ... >exclude_source = 1 I'm trying to create just one makefile, that will take the <target> as a param, so it can be used to build both i686 and ppc targets of TG (in this case). So, how would I specify that variance in just **one** ~/.pydistutils.cfg OR is there a way to have setuptools pickup different versions of this file, based on the <target> chosen for the makefile. Basically, I'll need something like: # i686 specific section [build] compiler = <native-gcc> [bdist_egg] plat_name = i686 exclude_source = 1 and, # ppc specific section [build] compiler = <cross-gcc> [bdist_egg] plat_name = ppc exclude_source = 1 Thanks, /venkat

2 1

cross compiling using setuptools
by Venkat Bommakanti Oct. 7, 2006

Oct. 7, 2006

Hi folks, I have a need to cross compile (i686 linux build system -> ppc linux target system) some py/C source pkgs: . RuleDispatch-0.5a0.dev-r2115.tar.gz . cElementTree-1.0.5-20051216.zip . PyProtocols-1.0a0dev_r2082.zip . Cheetah-2.0rc7.tar.gz to generate target-specific eggs... Actually, all are a part of TurboGears (TG). I was able to download all the comps to a local directory (i686 linux with python 2.4.2 compiled from sources, w/zlib enabled). Now, . while in the download dir (curr-dir - .), . with PYTHONPATH that includes "<my-i686-root>/usr/lib/python2.4/site-packages", and . with a command like: "easy_install --prefix=<my-i686-root>/usr -s <my-i686-root>/usr/bin -f . TurboGears" I'm able to build the native i686 target and deploy the i686 eggs to : <my-i686-root>/usr/lib/python2.4/site-packages just fine... No problem there. Assuming I can use setuptools as-is and "easy_install ..." invoke syntax like above - for the cross build (ppc target), how would I specify the options meant for: . setup.py (of each of the above comps): . -c (cross-gcc) . bdest_egg (setuptools): . --dist-dir . --plat-name . exclude-source in the "easy_install ..." command. I read: http://peak.telecommunity.com/DevCenter/setuptools#bdist-egg-create-a-pytho… but its unclear how the above options are passed on to "easy_install ...". I realize that I'd have to use a PYTHONPATH that includes something like: "<my-ppc-root>/usr/lib/python2.4/site-packages" for the ppc-target, so it doesn't complain about the: "bad install directory or PYTHONPATH" error... Also, 'am assuming its not a problem to include a path to ppc-eggs in the PYTHONPATH setting, before invoking the "easy_install ..." command for the cross compile. As an aside, would also like to know how I could enable the creation of target-specific .pyc files, in such a cross-compile scenario. I was able to build python itself in a cross compile (i686 build system -> ppc target system) mode, but the .pyc it generates appear to be i686-specific ones and not ppc-ones... when deployed to the ppc-system, it complains about the "bad magic number" error... Much appreciate any help. Thanks. /venkat

2 1

Re: [Distutils] Transitioning a Python Library on a Shared Network Drive to eggs/setuptools
by Phillip J. Eby Oct. 5, 2006

Oct. 5, 2006

At 10:59 AM 10/5/2006 -0400, Alexander Michael wrote: >I want to avoid the additional network round-trips being caused by setuptools >looking for packages without throwing out the proverbial baby with the >bath water (i.e. ditching eggs entirely). > >Looking into it further, it appears that a fair portion of the overhead is >incurred by making the network directory a sitedir (with site.addsitedir). >The following alternate setup seems to work a little better. In a .pth >file in the local site-packages directory I read the list of egg pathnames >from the network drive and add these eggs to sys.path myself. So, you're doing site.addpackage() to load the easy-install.pth that's on the network drive? That makes sense. > That way, I don't need to make the remote packages directory a sitedir > just so the easy-install.pth can be read from it. This allows me to > control the available versions and eggs remotely, while minimizing > network access. I am willing to pay the cost of reading each package from > the network directory (as well as the package list) in order to achieve > transparent updating. Now getting a simple help message is three times > faster and almost tolerable. > >Nevertheless, it sounds like I will either need to cache the shared >library on each users computer or ditch eggs altogether in order to bring >performance back to acceptable levels. >Since caching could make performance even better than before, I will try >to set this up. If you're going to read the easy-install.pth from the network drive, you could actually take one more step and see whether any eggs are listed there that weren't before, and go ahead and copy them to the local machine. However, there's another possibility regarding what's happening that you might consider. If your original setup was installing eggs to an easy-install.pth, you should try installing eggs to the network drive in --multi-version mode, so that only programs that explicitly request those eggs will add them to sys.path. The site directory will only get read once that way, and Python won't try to read the zip directories of every single egg, which is probably what's happening now. For this to work, your scripts must be wrappers generated by setuptools, or you must explicitly use pkg_resources.require() to ask for the libraries you need. (Recursive dependency lookups are automatic, however.) I'd suggest you give this a try, as it's an out-of-the-box configuration but one that's likely to get closer to your previous performance or *maybe* even exceed it due to its more effective use of zip files. Here's what you should do: the code you now have reading easy-install.pth from the network, should just tack the directory on the *end* of sys.path. Ignore easy-install.pth altogether, and in fact you can remove it from the network drive, and in future use the -m argument to easy_install when putting eggs on the network drive, so that it doesn't put anything in the .pth file. Get your scripts to request dependencies explicitly, and you should then have the maximum possible performance for an egg-based setup, because the directory will be listed only once, and zip directories will only be read for the actually required eggs. >If I do decide to ditch eggs altogether, if someone gives me an egg, is >there a way I can "unpack" it as if I did a traditional distutils install? Yes; simply extract it, and rename the resulting EGG-INFO directory to originalname.egg-info/, where originalname is the name of the original egg file. This will give you a "single version, externally managed" egg, in the format that is used for RPM, bdist_wininst, and other "system packager" egg installs. >The context is a scientific data analysis environment in which a group of >user-developers (nearly everyone works in both roles) both write data >analysis tools and perform data analysis. The tools are ever and rapidly >evolving along with the analysis, so the transparent upgrading that occurs >by using a shared drive has been convenient. We work in individual SVN >checkouts. After testing and committing our changes, we install the update >to the shared drive where by everyone automatically gets the change and we >assure that everyone is in sync. And I suppose it's asking too much to run "setup.py develop" on the SVN checkout when you want to get updated versions? (Because you could configure it to copy eggs down from the network drive at the time, using "setup.py develop -af /path/to/eggs".) Just a thought, but I suppose if you just want the *tools* to be up to date whenever you run them... I'm just confused by the idea that in your shop, if I ran an analysis twice in a row without taking any special action, I might end up with different results. But oh well.

2 1