This is an easy couple of questions (I hope). 1) What is the correct location for "pure" python module installations? $(prefix)/lib/python$VERSION/site-packages or $(prefix)/lib/site-python I believe most packages tend to install in site-packages but site-python makes more sense since I usually write my modules to deal with differences in python versions. 2) where should python shared libraries be placed if you want to have one install of python per site which includes different architectures? (For example, we may distribute our software on both SGI and Linux boxes.) As I understand it now, the python specific .so files need to be on the PYTHONPATH, which currently contains no information about the specific platform. The only information available is from sys.platform (along with os.uname) and that doesn't appear sufficient to distinguish between different SGI binary interfaces. Specifically, SGIs have 3 interfaces: "old" 32, "new" 32 and 64. These are specified during compilation by the environment variable SGI_ABI or by the command-line options (-32/-o32, -n32, -64). In theory we could compile libraries for each of the different interfaces, though we only support o32 at present. To do it fully we would have to write a wrapper script which runs the specified ABI version of Python ... oh, but then we could modify the PYTHONPATH to reflect the differences. Has there been a proposal for how to distribute/manage/install distributions with multiple architectures? The best I can think of for now is to modify site.py to add os.path.join(prefix, "lib", "python" + sys.version[:3], "site-packages", sys.platform), to the sitedirs list. Does this make sense, and should it be added to the 1.5.2 (or discussed more on c.l.py)? Andrew dalke@bioreason.com BTW, here's how SGI's java, which is only for 32 bits, manages things. "java" is actuall a shell script containing the following three snippets: # use -n32 binaries by default if [[ $SGI_ABI = -32 ]] then export JAVA_N32=0 elif [[ $SGI_ABI = -o32 ]] then export JAVA_N32=0 else export JAVA_N32=1 fi case $a in -32) JAVA_N32=0 shift ;; -o32) JAVA_N32=0 shift ;; -n32) JAVA_N32=1 shift ;; if [ $JAVA_N32 = 1 ] then check_path $LD_LIBRARYN32_PATH "LD_LIBRARYN32_PATH" if [ -z "$LD_LIBRARYN32_PATH" ] then if [ -z "$LD_LIBRARY_PATH" ] then LD_LIBRARYN32_PATH=$JAVA_HOME/lib32/sgi/$THREADS_TYPE else check_path $LD_LIBRARY_PATH "LD_LIBRARY_PATH" LD_LIBRARYN32_PATH="$JAVA_HOME/lib32/sgi/$THREADS_TYPE:$LD_LIBRARY_PATH" fi else LD_LIBRARYN32_PATH="$JAVA_HOME/lib32/sgi/$THREADS_TYPE:$LD_LIBRARYN32_PATH" fi export LD_LIBRARYN32_PATH prog=$JAVA_HOME/bin32/sgi/${THREADS_TYPE}/${progname} else check_path $LD_LIBRARY_PATH "LD_LIBRARY_PATH" if [ -z "$LD_LIBRARY_PATH" ] then LD_LIBRARY_PATH=$JAVA_HOME/lib/sgi/$THREADS_TYPE else LD_LIBRARY_PATH="$JAVA_HOME/lib/sgi/$THREADS_TYPE:$LD_LIBRARY_PATH" fi export LD_LIBRARY_PATH prog=$JAVA_HOME/bin/sgi/${THREADS_TYPE}/${progname} fi
Andrew Dalke writes:
1) What is the correct location for "pure" python module installations?
$(prefix)/lib/python$VERSION/site-packages or $(prefix)/lib/site-python
site-python/ is the original location, but never had a platform-specific counterpart. site-packages/ was intended to solve this, which is why it comes in both flavors, and deals with version- related issues. In general, 100% Python packages should be installed in site-packages/, because not all packages are sure to work across major version changes: it's too easy to rely on the bahavior of bugs in the library. ;-( This results in the safest (most conservative) installation and allows for faster detection on libraries which are not available for the Python version: ImportError is harder to misinterpret than buggy behavior. Only use site-python/ if you really are confident that your package will stand the test of Python interpreter updates. Unless there is a major problem with diskspace, just don't do this! Packages that contain native code should install in $(exec_prefix)/lib/python$VERSION/site-packages/.
As I understand it now, the python specific .so files need to be on the PYTHONPATH, which currently contains no information about
This varies by platform, but I don't think the various binary
flavors for SGIs are indicated; the directories on sys.path which are
platform-specific are either computed from $exec_prefix or contain
Python modules under $prefix that are only meaningful for the
platform.
I presume each of these platform variations would get a different
$exec_prefix, or the distinction would be hidden at a lower level
(such as mounting a filesystem at a common point based on binary
flavor; not hard with NFS).
-Fred
--
Fred L. Drake, Jr.
Fred Drake said:
In general, 100% Python packages should be installed in site-packages/,
Okay, I'll follow that guideline.
I presume each of these platform variations would get a different $exec_prefix, or the distinction would be hidden at a lower level (such as mounting a filesystem at a common point based on binary flavor; not hard with NFS).
Thinking about your comment some more, my question is much less of an issue than I had thought. If we (Bioreason) distribute binary libraries and want to be specific to the given architecture we can install under the existing python tree, or somewhere else. If we install under the existing "site-packages" directory, then the sysadmin has already figured out how to handle the differences between OSes (eg, by having distinct install or with the NFS mount trick). And there's no way we can predict that method. If we distribute as our own directory tree, then the PYTHONPATH will already have to be modified, so we can say: Add "source /blah/bling/blang/blang.csh" to your ".cshrc" and then define "blang.csh" as something like: setenv OUR_PACKAGE /blah/bling/blang set arch=`$OUR_PACKAGE/getarch.sh` set libdir=$OUR_PACKAGE/lib/$arch if ${?PYTHONPATH} then setenv PYTHONPATH $PYTHONPATH:$OUR_PACKAGE/python:$libdir else setenv PYTHONPATH $OUR_PACKAGE/python:$libdir endif The only time this is an issue is if the sysadmin doesn't already have a mechanism for installing multiple architectures, and there's no way that can be mandated. (For example, I think we'll end up internally choosing libraries by setting up our own PYTHONPATH as needed for each architecture, and we'll be able to specify our own criterion to distinguish between then in ways that the normal Python install *cannot* discern.) Andrew dalke@bioreason.com
Quoth Fred L. Drake, on 29 March 1999:
site-python/ is the original location, but never had a platform-specific counterpart. site-packages/ was intended to solve this, which is why it comes in both flavors, and deals with version- related issues. In general, 100% Python packages should be installed in site-packages/, because not all packages are sure to work across major version changes: it's too easy to rely on the bahavior of bugs in the library. ;-( This results in the safest (most conservative) installation and allows for faster detection on libraries which are not available for the Python version: ImportError is harder to misinterpret than buggy behavior.
I've sorta been wondering about that myself -- I've had to resort to dissecting Python Makefiles, doing multiple test installations with slight parameter tweaks, and taking careful notes on the whole process to try to figure out what's happening and what Distutils should try to emulate/enforce/whatever. One thing that bugs me: there doesn't seem to be an elegant way to have multiple sub-versions of Python installed on the same machine, eg. 1.5.1 and 1.5.2b2 (or whatever the latest alpha/beta is at a given time). I have taken to setting prefix=/usr/local/python-1.5.1 and prefix=/usr/local/python-1.5.2b2 respectively, and screwing around with symlinks in /usr/local/bin to get things the way I want them. Is this the best anyone has come up with? It would be nice to subdivide /usr/local/lib/python1.5, but the fact that Python figures out sys.path at runtime seems to preclude this. Are my impressions correct? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
participants (3)
-
Andrew Dalke -
Fred L. Drake -
Greg Ward