Hi all..
Firstly, I would like to say thanks for the setuptools package, which
I was introduced to after reading about the RuleDispatch package on
the IBM developerworks charming python series. Oh btw. RuleDispatch is
the most useful python package that I have seen in the last year in
the python world, so thanks for that also ;)
Now that the congrats and hugs are out of the way, I would like to ask
a question. How can I tell setuptools not to put packages, such as
dispatch, protocols, and setuptools, in a subdirectory named the same
as the egg and to just put the package name.
For example from pydoc I currently get:
c:\python\lib\site-packages\ruledispatch-0.5a0.dev_r2100-py2.4-win32.egg
dispatch (package)
c:\python\lib\site-packages\pyprotocols-1.0a0dev_r2082-py2.4-win32.egg
dispatch
protocols (package)
c:\python\lib\site-packages\setuptools-0.6a8-py2.4.egg
easy_install
pkg_resources
setuptools (package)
site
what I would like to see is just:
..\site-packages
dispatch (package)
..\site-packages
protocols (package)
..\site-packages
setuptools (package)
Maybe it is just a pet-peeve, but I like to keep a nice tidy
site-packages directory, and these long directory names just seem to
me to be pollution of my site-packages directory in my command shell.
If there is an option to have just the packages, I would really
appreciate someone telling me what it is. If not maybe it could be
considered, and to put whatever meta-data the directory names are
providing somewhere else..
Thanks,
Anthony
Hi,
While compiling some py/C source pkgs, such as:
. RuleDispatch-0.5a0.dev-r2115.tar.gz
. cElementTree-1.0.5-20051216.zip
. PyProtocols-1.0a0dev_r2082.zip
. Cheetah-2.0rc7.tar.gz
I noticed in the logs that each (after being unpacked) has
the following command invoked:
setup.py -q bdist_egg --dist-dir /tmp/....
Since selecting a 'compiler' is not an option for the
"bdist_egg" command, are even the settings in
~/.pydistutils.cfg file:
[build]
compiler = unix
[bdist_egg]
plat_name = ppc
picked up ?
They appear not to be picked up, and I'm suspecting that
since the [build] settings are valid only for the "build" command,
it will not have any effect on the "bdist_egg" command used
by all the above comps. Is that accurate ?
If plat_name is not picked up, how is setuptools figuring
out that the platform is i686 (given that the build is being
done for a i686 linux box). Verbose logs don't show where/how
that decision is made...
Thanks,
/venkat
>>Based on this suggestion, for the cross compile case, I tried to set:
>> [bdist_egg]
>> plat_name = ppc
>>but, it still insists on falling back to i686 for plat_name. Could be I'm
>>doing something fundamentally wrong.
>Dunno. Is this in $HOME/.pydistutils.cfg?
For the cross compile case (i686 build box -> ppc target), rather than
the setup.py/setup.cfg option in the TurboGears src/pkg dir, I thought
I'll try the basic method of using $HOME/.pydistutils.cfg first. So I
created one ($HOME/.pydistutils.cfg) with the following contents:
[build]
compiler=powerpc-unknown-linux-gcc
[bdist_egg]
pla_name=ppc
When I run:
easy_install --prefix=usr -s usr/bin -f . TG.egg
I get an error msg:
error: Setup script exited with error: don't know how to compile C/C++
code on platform 'posix' with 'powerpc-unknown-linux-gcc' compiler
I have 'powerpc-unknown-linux-gcc' in my path. Could it be that its not
picking up some include dirs etc.
I also tried the same with i686 build box -> i686 target, pointing to my
i686 gcc 'i686-pc-linux-gcc'...
[build]
compiler=i686-pc-linux-gcc
[bdist_egg]
pla_name=i686
I get the same error...
error: Setup script exited with error: don't know how to compile C/C++
code on platform 'posix' with 'i686-pc-linux-gcc' compiler
Any idea what may be wrong here...
Thanks much,
/venkat
>>Based on this suggestion, for the cross compile case, I tried to set:
>> [bdist_egg]
>> plat_name = ppc
>>but, it still insists on falling back to i686 for plat_name. Could be I'm
>>doing something fundamentally wrong.
>Dunno. Is this in $HOME/.pydistutils.cfg?
I'm actually using a seperate <temp_dir> for each <compile_target>.
So, they are:
<top-of-src-tree>/temp_i686/setup.cfg
and
<top-of-src-tree>/temp_ppc/setup.cfg
They have their own (temp) set of TG comps in their native tar/zip
formats. This will allow for a developer to snap a tree and build from
there, without any build-specifc files in their $HOMEs. Each will have
<top-of-src-tree>/temp_<tgt>/easy_install -f . TurboGears
invoked from their respective directories, and will use their respective
setup.cfg files.
>> 'am also
>>assuming the syntax is the same for setup.cfg (local to TG source dir) and
>>pydistutils.cfg ($HOME).
>On Unix-like platforms, it's ".pydistutils.cfg" (note the '.' at the
>beginning), NOT "pydistutils.cfg". The latter name is used only on Windows.
Based on the above layout, and from what I've understood so far,
I'm assuming:
<top-of-src-tree>/temp_<tgt>/setup.cfg
is the equivalent of:
$HOME/.pydistutils.cfg
per the responses so far, and notes in pages like:
http://docs.python.org/inst/config-syntax.html
Is is accurate to say that the contents of setup.cfg & pydistutils.cfg are the
exactly the same ?
Thanks a lot,
/venkat
>>Assuming I can use setuptools as-is and "easy_install ..." invoke syntax
>>like above - for the cross build (ppc target), how would I specify the
>>options meant for:
>> . setup.py (of each of the above comps):
>> . -c (cross-gcc)
>> . bdest_egg (setuptools):
>> . --dist-dir
>> . --plat-name
>> . exclude-source
>>in the "easy_install ..." command. I read:
>Put them in your ~/.pydistutils.cfg file, using e.g.
>[build]
>compiler = ...
>[bdist_egg]
>plat_name = ...
>exclude_source = 1
Based on this suggestion, for the cross compile case, I tried to set:
[bdist_egg]
plat_name = ppc
but, it still insists on falling back to i686 for plat_name. Could be I'm
doing something fundamentally wrong.
'am also trying to "set" as many params to easy_install / setup.py-of-comp
(in TG's suite of components) as possible this way... but its unclear if
I'm doing them right...
Is there a reference somewhere in (http://peak.telecommunity.com/) to all
. the sections that can go into these files, and
. the key/value pairs of settings, that can go into each section.
For example, I couldn't see anywhere it documented that [bdist_egg] is one
of the many valid sections in these files. I'm not doubting the earlier statement,
but just unable to find a single place that lists all the possible cases. 'am also
assuming the syntax is the same for setup.cfg (local to TG source dir) and
pydistutils.cfg ($HOME).
Thanks,
/venkat
>At 02:24 PM 10/8/2006 -0700, Venkat Bommakanti wrote:
>> Wondering if a "setup.cfg" for this component (TG) in the component's
>> directory itself, will serve the purpose.
>Well, if each project is going to be built for only one platform,
>sure. But the setup.cfg *overrides* the .pydistutils.cfg, so you don't
>want to distribute a bunch of site/platform-specific options in the
>project's published setup.cfg.
>In any case, I thought your original issue had to do with getting
>easy_install to handle these packages automatically, without modifying the
>original packages in any way. And that's the context in which I've been
>answering your questions.
I apologize for the confusion here... Will try to clarify the full need...
Basically, the need is such:
. a developer would snap a <src> tree
. build python <target>, TG for a <target> of choice (i686/ppc)
A developer could potentially fire the build of both targets simultaneously.
And to build the ppc python/TG (cross target) they'd use the necessary
i686 python that is part of a <tools> directory on the i686 build box.
Under those circumstances, and also the desire to keep all build related
files (incuding .pydistutils.cfg or equivalents) within the source tree itself,
I wanted to know the best recourse.
Using $HOME/.pydistutils.cfg may not be a viable option due the fact
that it'd reside outside the <snapped> source tree, and could become a
maintenance issue... Would like a solution where one gets the cfg file
(or equivalent) when they snap a tree direct...
I was hoping something like:
<top-of-src-tree>/TurboGears/<pydistutils.cfg or setup.cfg equivalent>
with info relevant to:
[build]
compiler = $COMPILER (native or cross-gcc)
[bdist_egg]
plat_name = $TARGET (i686 or ppc)
exclude_source = 1
would let me specify (target-based) info that is picked up according
to the build target.
Thanks again,
/venkat
>At 02:07 PM 10/8/2006 -0700, Venkat Bommakanti wrote:
>>So, how would I specify that variance in just **one** ~/.pydistutils.cfg
>>OR is there a way to have setuptools pickup different versions of this
>>file, based on the <target> chosen for the makefile.
>You could use different values of $HOME, since that's where the per-user
>config file is read from.
Wondering if a "setup.cfg" for this component (TG) in the component's
directory itself, will serve the purpose. That ways, each developer
wouldn't have maintain thier own ~/.pydistutils.cfg files.
Thanks again,
/venkat
Thanks Phillip.
Have a followup question below.
/venkat
>>Assuming I can use setuptools as-is and "easy_install ..." invoke syntax
>>like above - for the cross build (ppc target), how would I specify the
>>options meant for:
>> . setup.py (of each of the above comps):
>>
>> . -c (cross-gcc)
>> . bdest_egg (setuptools):
>> . --dist-dir
>> . --plat-name
>> . exclude-source
>>in the "easy_install ..." command. I read:
>>
>>http://peak.telecommunity.com/DevCenter/setuptools#bdist-egg-create-a-pytho…
>>but its unclear how the above options are passed on to "easy_install ...".
>Put them in your ~/.pydistutils.cfg file, using e.g.
>[build]
>compiler = ...
>[bdist_egg]
>plat_name = ...
>exclude_source = 1
I'm trying to create just one makefile, that will take the <target> as a param, so it can be used to build both i686 and ppc targets of TG (in this case). So, how would I specify that variance in just **one** ~/.pydistutils.cfg OR is there a way to have setuptools pickup different versions of this file, based on the <target> chosen for the makefile.
Basically, I'll need something like:
# i686 specific section
[build]
compiler = <native-gcc>
[bdist_egg]
plat_name = i686
exclude_source = 1
and,
# ppc specific section
[build]
compiler = <cross-gcc>
[bdist_egg]
plat_name = ppc
exclude_source = 1
Thanks,
/venkat
Hi folks,
I have a need to cross compile (i686 linux build system -> ppc linux target system) some py/C source pkgs:
. RuleDispatch-0.5a0.dev-r2115.tar.gz
. cElementTree-1.0.5-20051216.zip
. PyProtocols-1.0a0dev_r2082.zip
. Cheetah-2.0rc7.tar.gz
to generate target-specific eggs... Actually, all are a part of TurboGears (TG).
I was able to download all the comps to a local directory (i686 linux with python 2.4.2 compiled from sources, w/zlib enabled). Now,
. while in the download dir (curr-dir - .),
. with PYTHONPATH that includes "<my-i686-root>/usr/lib/python2.4/site-packages", and
. with a command like:
"easy_install --prefix=<my-i686-root>/usr -s <my-i686-root>/usr/bin -f . TurboGears"
I'm able to build the native i686 target and deploy the i686 eggs to :
<my-i686-root>/usr/lib/python2.4/site-packages
just fine... No problem there.
Assuming I can use setuptools as-is and "easy_install ..." invoke syntax like above - for the cross build (ppc target), how would I specify the options meant for:
. setup.py (of each of the above comps):
. -c (cross-gcc)
. bdest_egg (setuptools):
. --dist-dir
. --plat-name
. exclude-source
in the "easy_install ..." command. I read:
http://peak.telecommunity.com/DevCenter/setuptools#bdist-egg-create-a-pytho…
but its unclear how the above options are passed on to "easy_install ...".
I realize that I'd have to use a PYTHONPATH that includes something like:
"<my-ppc-root>/usr/lib/python2.4/site-packages"
for the ppc-target, so it doesn't complain about the:
"bad install directory or PYTHONPATH"
error... Also, 'am assuming its not a problem to include a path to ppc-eggs in the PYTHONPATH setting, before invoking the "easy_install ..." command for the cross compile.
As an aside, would also like to know how I could enable the creation of target-specific .pyc files, in such a cross-compile scenario. I was able to build python itself in a cross compile (i686 build system -> ppc target system) mode, but the .pyc it generates appear to be i686-specific ones and not ppc-ones... when deployed to the ppc-system, it complains about the "bad magic number" error...
Much appreciate any help. Thanks.
/venkat
At 10:59 AM 10/5/2006 -0400, Alexander Michael wrote:
>I want to avoid the additional network round-trips being caused by setuptools
>looking for packages without throwing out the proverbial baby with the
>bath water (i.e. ditching eggs entirely).
>
>Looking into it further, it appears that a fair portion of the overhead is
>incurred by making the network directory a sitedir (with site.addsitedir).
>The following alternate setup seems to work a little better. In a .pth
>file in the local site-packages directory I read the list of egg pathnames
>from the network drive and add these eggs to sys.path myself.
So, you're doing site.addpackage() to load the easy-install.pth that's on
the network drive? That makes sense.
> That way, I don't need to make the remote packages directory a sitedir
> just so the easy-install.pth can be read from it. This allows me to
> control the available versions and eggs remotely, while minimizing
> network access. I am willing to pay the cost of reading each package from
> the network directory (as well as the package list) in order to achieve
> transparent updating. Now getting a simple help message is three times
> faster and almost tolerable.
>
>Nevertheless, it sounds like I will either need to cache the shared
>library on each users computer or ditch eggs altogether in order to bring
>performance back to acceptable levels.
>Since caching could make performance even better than before, I will try
>to set this up.
If you're going to read the easy-install.pth from the network drive, you
could actually take one more step and see whether any eggs are listed there
that weren't before, and go ahead and copy them to the local machine.
However, there's another possibility regarding what's happening that you
might consider. If your original setup was installing eggs to an
easy-install.pth, you should try installing eggs to the network drive in
--multi-version mode, so that only programs that explicitly request those
eggs will add them to sys.path. The site directory will only get read once
that way, and Python won't try to read the zip directories of every single
egg, which is probably what's happening now.
For this to work, your scripts must be wrappers generated by setuptools, or
you must explicitly use pkg_resources.require() to ask for the libraries
you need. (Recursive dependency lookups are automatic, however.)
I'd suggest you give this a try, as it's an out-of-the-box configuration
but one that's likely to get closer to your previous performance or *maybe*
even exceed it due to its more effective use of zip files.
Here's what you should do: the code you now have reading easy-install.pth
from the network, should just tack the directory on the *end* of
sys.path. Ignore easy-install.pth altogether, and in fact you can remove
it from the network drive, and in future use the -m argument to
easy_install when putting eggs on the network drive, so that it doesn't put
anything in the .pth file. Get your scripts to request dependencies
explicitly, and you should then have the maximum possible performance for
an egg-based setup, because the directory will be listed only once, and zip
directories will only be read for the actually required eggs.
>If I do decide to ditch eggs altogether, if someone gives me an egg, is
>there a way I can "unpack" it as if I did a traditional distutils install?
Yes; simply extract it, and rename the resulting EGG-INFO directory to
originalname.egg-info/, where originalname is the name of the original egg
file. This will give you a "single version, externally managed" egg, in
the format that is used for RPM, bdist_wininst, and other "system packager"
egg installs.
>The context is a scientific data analysis environment in which a group of
>user-developers (nearly everyone works in both roles) both write data
>analysis tools and perform data analysis. The tools are ever and rapidly
>evolving along with the analysis, so the transparent upgrading that occurs
>by using a shared drive has been convenient. We work in individual SVN
>checkouts. After testing and committing our changes, we install the update
>to the shared drive where by everyone automatically gets the change and we
>assure that everyone is in sync.
And I suppose it's asking too much to run "setup.py develop" on the SVN
checkout when you want to get updated versions? (Because you could
configure it to copy eggs down from the network drive at the time, using
"setup.py develop -af /path/to/eggs".) Just a thought, but I suppose if
you just want the *tools* to be up to date whenever you run them... I'm
just confused by the idea that in your shop, if I ran an analysis twice in
a row without taking any special action, I might end up with different
results. But oh well.
