I have been building/installing 3rd party python (numeric/scipy/etc.) as a user instead of as root lately. This means the packages are installed in a non-standard place (like /home/eric/linux or /home/eric/sunos). I do this using: python setup.py install --prefix=$OSDIR where OSDIR is an environment variable set to specify a platform dependent path (my home directory is shared between about 6 different architectures). The user (instead of root) installed locations have caused several annoyances, one with include directories and one with .pth files. They have both manifested themselves when dealing with Numeric. (1) include directories. Distutils knows to include files from /usr/include/python2.2 (or wherever it is installed) whenever building extension modules. Numeric installs its header files inside this directory when installed as root. However, when I install Numeric in /home/eric/linux, the header files are installed in /home/eric/linux/python2.2. Distutils doesn't know to look in hear for headers. To solve this, I'd have to hack all the setup.py files for modules that rely on Numeric to use my include_dirs. This isn't so nice. To rectify the problem, scipy_distutils now searches for an environment variable called PYTHONINCLUDE that uses the same syntax as PYTHONPATH and PATH. If PYTHONINCLUDE is found, the directories listed in it are added to the include path during compilation of C/C++ files by build_ext.py and build_clib.py. On my machine, I specify export PYTHONINCLUDE=$OSDIR/include/python2.2 and all setup.py files work unaltered. I think this is a reasonable solution. Has anyone solved this problem with an alternative method that might be better. (2) .pth files Numeric is a "package" that doesn't really behave as a package. It dumps all of its files in site-packages/Numeric, but it doesn't define a __init__.py file. Instead, it also puts a Numeric.pth file in site-packages that tells python to also search site-packages/Numeric for modules. This works fine when Numeric is installed as root, but it doesn't work when it is installed in non-standard places. This is because .pth files are only expaneded when they exist in the "system" locations. They are ignored anywhere else in the search path. As a result, after installing Numeric with "python setup.py install --prefix=$OSDIR", importing Numeric fails even if PYTHONPATH=$OSDIR/lib/python2.2/site-packages The fix is easy for someone who understands what is going on -- add $OSDIR/lib/python2.2/site-packages/Numeric to the PYTHONPATH. However, the uninitiated may flail around a long time trying to figure out why installing a package to there standard location doesn't work -- especially if it has worked for every other package they have installed. I haven't figured out a nice work around here, other than actually making Numeric a real package which I doubt is even close to an option. Does anyone else have a better solution? Thanks, eric ---------------------------------------------- eric jones 515 Congress Ave www.enthought.com Suite 1614 512 536-1057 Austin, Tx 78701
"EJ" == eric jones <eric@enthought.com> writes:
EJ> (2) .pth files Numeric is a "package" that doesn't really EJ> behave as a package. It dumps all of its files in EJ> site-packages/Numeric, but it doesn't define a __init__.py EJ> file. Instead, it also puts a Numeric.pth file in EJ> site-packages that tells python to also search EJ> site-packages/Numeric for modules. This works fine when EJ> Numeric is installed as root, but it doesn't work when it is EJ> installed in non-standard places. This is because .pth files EJ> are only expaneded when they exist in the "system" locations. EJ> They are ignored anywhere else in the search path. As a EJ> result, after installing Numeric with "python setup.py install EJ> --prefix=$OSDIR", importing Numeric fails even if I guess you must have thought of this but perhaps Python's site.py can be modified to look for .pth files inside any directory on the PYTHONPATH. If I understand correctly, this should solve the problem? prabhu
Eric Jones writes:
(2) .pth files Numeric is a "package" that doesn't really behave as a package. It dumps all of its files in site-packages/Numeric, but it doesn't define a __init__.py file. Instead, it also puts a Numeric.pth file in site-packages that tells python to also search site-packages/Numeric for modules. This works fine when Numeric is installed as root, but it doesn't work when it is installed in non-standard places. This is because .pth files are only expaneded when they exist in the "system" locations. They are ignored anywhere else in the search path. As a result, after installing Numeric with "python setup.py install --prefix=$OSDIR", importing Numeric fails even if
PYTHONPATH=$OSDIR/lib/python2.2/site-packages
The fix is easy for someone who understands what is going on -- add $OSDIR/lib/python2.2/site-packages/Numeric to the PYTHONPATH. However, the uninitiated may flail around a long time trying to figure out why installing a package to there standard location doesn't work -- especially if it has worked for every other package they have installed.
Something we have done locally at STScI is to hack a sitecustomize.py file to do what someone else suggested, i.e., to look at all directories in the pythonpath for .pth files. Our version of this file then looks to see if it has masked any other sitecustomize.py files in the path and executes the next one (so as not to screw up existing sitecustomize.py files). Is this a possible solution?
I haven't figured out a nice work around here, other than actually making Numeric a real package which I doubt is even close to an option. Does anyone else have a better solution?
Well, we were going to ask about that eventually (for numarray). This spurred me to raise it now. Should numarray use the package mechanism? This would mean that one would do things like from numarray.FFT import * or from numarray import FFT The advantages are that numarray 3rd party modules or add-ons are more cleanly segregated from the numarray files, and it eliminates the possiblity of name collisions with other modules not related to numarray. Currently we have a problem with the 3rd party modules in that we have to give them a different name than the equivalent versions for Numeric (because of name collisions). Thus FFT becomes FFT2 which is not a very good solution. Of course this change impacts current code since the imports would have to be changed (generally not a hard code change, but a annoyance nonetheless). Adding the subdirectories to the path would make it work for old code (wouldn't it?). If we are going to change it, now's the time to consider it. Perry
Well, we were going to ask about that eventually (for numarray). This spurred me to raise it now. Should numarray use the package mechanism?
Absolutely! I've always found it extremely irritating that Numeric isn't a 'true' package, considering how the __init__.py package mechanism is considered fairly standard these days. The .pth hackery is always a PITA and can easily be an unsurmountable wall for a not too experienced user who needs a slightly non-standard setup. I would definitely be +1 (+10 if such a thing is accepted :) on making Numarray (and even Numeric's new versions, if possible) a real package. Cheers, f.
-----Original Message----- From: scipy-dev-admin@scipy.net [mailto:scipy-dev-admin@scipy.net] On Behalf Of Perry Greenfield Sent: Thursday, December 19, 2002 11:12 AM To: scipy-dev@scipy.net Cc: jmiller@stsci.edu Subject: RE: [SciPy-dev] issues with installing Numeric in a user directory
Eric Jones writes:
(2) .pth files Numeric is a "package" that doesn't really behave as a package. It dumps all of its files in site-packages/Numeric, but it doesn't define a __init__.py file. Instead, it also puts a Numeric.pth file in site-packages that tells python to also search site-packages/Numeric for modules. This works fine when Numeric is installed as root, but it doesn't work when it is installed in non-standard places. This is because .pth files are only expaneded when they exist in the "system" locations. They are ignored anywhere else in the search path. As a result, after installing Numeric with "python setup.py install --prefix=$OSDIR", importing Numeric fails even if
PYTHONPATH=$OSDIR/lib/python2.2/site-packages
The fix is easy for someone who understands what is going on -- add $OSDIR/lib/python2.2/site-packages/Numeric to the PYTHONPATH. However, the uninitiated may flail around a long time trying to figure out why installing a package to there standard location doesn't work -- especially if it has worked for every other package they have installed.
Something we have done locally at STScI is to hack a sitecustomize.py file to do what someone else suggested, i.e., to look at all
in the pythonpath for .pth files. Our version of this file then looks to see if it has masked any other sitecustomize.py files in the path and executes the next one (so as not to screw up existing sitecustomize.py files). Is this a possible solution?
I haven't figured out a nice work around here, other than actually making Numeric a real package which I doubt is even close to an
+1 on numarray as a package. eric ---------------------------------------------- eric jones 515 Congress Ave www.enthought.com Suite 1614 512 536-1057 Austin, Tx 78701 directories option.
Does anyone else have a better solution?
Well, we were going to ask about that eventually (for numarray). This spurred me to raise it now. Should numarray use the package mechanism? This would mean that one would do things like
from numarray.FFT import *
or
from numarray import FFT
The advantages are that numarray 3rd party modules or add-ons are more cleanly segregated from the numarray files, and it eliminates the possiblity of name collisions with other modules not related to numarray. Currently we have a problem with the 3rd party modules in that we have to give them a different name than the equivalent versions for Numeric (because of name collisions). Thus FFT becomes FFT2 which is not a very good solution.
Of course this change impacts current code since the imports would have to be changed (generally not a hard code change, but a annoyance nonetheless). Adding the subdirectories to the path would make it work for old code (wouldn't it?). If we are going to change it, now's the time to consider it.
Perry _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.net http://www.scipy.net/mailman/listinfo/scipy-dev
Eric Jones writes:
(2) .pth files Numeric is a "package" that doesn't really behave as a package. It dumps all of its files in site-packages/Numeric, but it doesn't define a __init__.py file. Instead, it also puts a Numeric.pth file in site-packages that tells python to also search site-packages/Numeric for modules. This works fine when Numeric is installed as root, but it doesn't work when it is installed in non-standard places. This is because .pth files are only expaneded when they exist in the "system" locations. They are ignored anywhere else in the search path. As a result, after installing Numeric with "python setup.py install --prefix=$OSDIR", importing Numeric fails even if
PYTHONPATH=$OSDIR/lib/python2.2/site-packages
The fix is easy for someone who understands what is going on -- add $OSDIR/lib/python2.2/site-packages/Numeric to the PYTHONPATH. However, the uninitiated may flail around a long time trying to figure out why installing a package to there standard location doesn't work -- especially if it has worked for every other package they have installed.
Something we have done locally at STScI is to hack a sitecustomize.py file to do what someone else suggested, i.e., to look at all directories in the pythonpath for .pth files. Our version of this file then looks to see if it has masked any other sitecustomize.py files in the path and executes the next one (so as not to screw up existing sitecustomize.py files). Is this a possible solution?
Yes. It is a hack IMHO also. It trades environment variable changes to site.py changes -- I'd rather require neither. Still, I'd like to see what you've done. Can you send me that file?
I haven't figured out a nice work around here, other than actually making Numeric a real package which I doubt is even close to an
option.
Does anyone else have a better solution?
Well, we were going to ask about that eventually (for numarray). This spurred me to raise it now. Should numarray use the package mechanism? This would mean that one would do things like
from numarray.FFT import *
or
from numarray import FFT
The advantages are that numarray 3rd party modules or add-ons are more cleanly segregated from the numarray files, and it eliminates the possiblity of name collisions with other modules not related to numarray. Currently we have a problem with the 3rd party modules in that we have to give them a different name than the equivalent versions for Numeric (because of name collisions). Thus FFT becomes FFT2 which is not a very good solution.
Of course this change impacts current code since the imports would have to be changed (generally not a hard code change, but a annoyance nonetheless). Adding the subdirectories to the path would make it work for old code (wouldn't it?). If we are going to change it, now's the time to consider it.
I think now is the time to change it. eric
Yes. It is a hack IMHO also. It trades environment variable changes to site.py changes -- I'd rather require neither. Still, I'd like to see what you've done. Can you send me that file?
I'll try to do that tomorrow. By the way, I'm not talking about site.py. The file sitecustomize.py can be anywhere in your path if I remember correctly, and does not require changing site.py. The primary reason we use this is so that we can support running multiple versions of Python without messing with a user's pythonpath. The script checks to see which version of python you are running, and then dynamically adds the appropriate directories to the path (since different versions often require different extension modules). We put 2.0, 2.1, 2.2, 2.2.2 directories for our modules in one directory and the parent directory is what users must put in their path. A sitecustomize.py in that directory does the rest (Along with .pth files). Since we don't have privileges for the site-packages directory, we wanted to have a directory tree we had control over, but where we didn't burden the user having to deal with version and multple paths to add for each of the modules we had.
I haven't figured out a nice work around here, other than actually making Numeric a real package which I doubt is even close to an
option.
Does anyone else have a better solution?
Of course this change impacts current code since the imports would have to be changed (generally not a hard code change, but a annoyance nonetheless). Adding the subdirectories to the path would make it work for old code (wouldn't it?). If we are going to change it, now's the time to consider it.
I think now is the time to change it.
eric
Let me raise the issue on the numpy list also, but it sure seems like the right thing to do. Perry
participants (4)
-
eric jones -
Fernando Perez -
Perry Greenfield -
Prabhu Ramachandran