On Thu, 19 Dec 2002 eric@localhost.localdomain wrote:
Modified Files: build_clib.py build_ext.py
Actually, build_clib.py does not need the PYTHONINCLUDE feature. I'd remove it.
Log Message: made changes to use a PYTHONINCLUDE environment variable to search for include files during extension building.
(1) include directories. Distutils knows to include files from /usr/include/python2.2 (or wherever it is installed) whenever building extension modules. Numeric installs its header files inside this directory when installed as root. However, when I install Numeric in /home/eric/linux, the header files are installed in /home/eric/linux/python2.2. Distutils doesn't know to look in hear for headers. To solve this, I'd have to hack all the setup.py files for modules that rely on Numeric to use my include_dirs. This isn't so nice.
To rectify the problem, scipy_distutils now searches for an environment variable called PYTHONINCLUDE that uses the same syntax as PYTHONPATH and PATH. If PYTHONINCLUDE is found, the directories listed in it are added to the include path during compilation of C/C++ files by build_ext.py and build_clib.py. On my machine, I specify
export PYTHONINCLUDE=$OSDIR/include/python2.2
and all setup.py files work unaltered. I think this is a reasonable solution.
There is one gotcha with the implementation of this solution. Consider the following situation: Numeric is installed as root. An user installs another version of Numeric using, say, --prefix=$HOME. As a result Numeric header files are installed to $HOME/include/python2.2. Now, when building an extension module that uses Numeric and defining PYTHONINCLUDE=$HOME/include/python2.2, then the compiler argument will contain -I/usr/include/python2.2 -I$HOME/include/python2.2 As a result, wrong Numeric header files (from /usr/include/python2.2/Numeric) are being used when compiling extension modules. Obviously, $HOME/include/python2.2 should appear before /usr/include/python2.2 and that can be easily accomplished by inserting PYTHONINCLUDE directory(-ies) just before the '/usr/include/python2.2' item in self.include_dirs list. It appears from distutils code that '/usr/include/python2.2' is always the last item in the list and that makes the fix easier. However, IMHO distutils should take care of inserting <prefix>/include/python2.2 to the list include_dirs whenever --prefix=<prefix> is used. And if this will be fixed in distutils, then PYTHONINCLUDE feature becomes YAGNI. So, I would avoid introducing PYTHONINCLUDE feature because the original issue can be resolved using sitecustomize.py alone (after all that is the "standard" way to customize Python). Moreover, the other issue of adding <prefix>/lib/python<version>/site-packages (and .pth extensions) to sys.path can be resolved in sitecustomize.py as well as Perry already indicated (and users would not need to modify their PYTHONPATH). A version of sitecustomize.py is attached to this message and I'd like to propose adding this script to scipy directory. The attached sitecustomize.py script contains also a fix to distutils.util.get_platform function that is broken for Mac OSX system. This script might be useful also for back-porting other Python fixes/features to older Python versions that SciPy is supposed to support. What do you think? Pearu
-----Original Message----- From: scipy-dev-admin@scipy.net [mailto:scipy-dev-admin@scipy.net] On Behalf Of Pearu Peterson Sent: Saturday, December 28, 2002 6:18 PM To: scipy-dev@scipy.org Subject: [SciPy-dev] PYTHONINCLUDE and sitecustomize.py
On Thu, 19 Dec 2002 eric@localhost.localdomain wrote:
Modified Files: build_clib.py build_ext.py
Actually, build_clib.py does not need the PYTHONINCLUDE feature. I'd remove it.
Log Message: made changes to use a PYTHONINCLUDE environment variable to search for include files during extension building.
(1) include directories. Distutils knows to include files from /usr/include/python2.2 (or wherever it is installed) whenever building extension modules. Numeric installs its header files inside this directory when installed as root. However, when I install Numeric in /home/eric/linux, the header files are installed in /home/eric/linux/python2.2. Distutils doesn't know to look in hear for headers. To solve this, I'd have to hack all the setup.py files for modules that rely on Numeric to use my include_dirs. This isn't so nice.
To rectify the problem, scipy_distutils now searches for an environment variable called PYTHONINCLUDE that uses the same syntax as PYTHONPATH and PATH. If PYTHONINCLUDE is found, the directories listed in it are added to the include path during compilation of C/C++ files by build_ext.py and build_clib.py. On my machine, I specify
export PYTHONINCLUDE=$OSDIR/include/python2.2
and all setup.py files work unaltered. I think this is a reasonable solution.
There is one gotcha with the implementation of this solution. Consider the following situation:
Numeric is installed as root. An user installs another version of Numeric using, say, --prefix=$HOME. As a result Numeric header files are installed to $HOME/include/python2.2. Now, when building an extension module that uses Numeric and defining PYTHONINCLUDE=$HOME/include/python2.2, then the compiler argument will contain -I/usr/include/python2.2 -I$HOME/include/python2.2 As a result, wrong Numeric header files (from /usr/include/python2.2/Numeric) are being used when compiling extension modules.
Obviously, $HOME/include/python2.2 should appear before /usr/include/python2.2 and that can be easily accomplished by inserting PYTHONINCLUDE directory(-ies) just before the '/usr/include/python2.2' item in self.include_dirs list. It appears from distutils code that '/usr/include/python2.2' is always the last item in the list and that makes the fix easier.
However, IMHO distutils should take care of inserting <prefix>/include/python2.2 to the list include_dirs whenever --prefix=<prefix> is used. And if this will be fixed in distutils,
PYTHONINCLUDE feature becomes YAGNI.
So, I would avoid introducing PYTHONINCLUDE feature because the original issue can be resolved using sitecustomize.py alone (after all that is
"standard" way to customize Python). Moreover, the other issue of adding <prefix>/lib/python<version>/site-packages (and .pth extensions) to sys.path can be resolved in sitecustomize.py as well as Perry already indicated (and users would not need to modify
Hey Pearu, The script wasn't attached. Is this a mailing list issue, or just an oversight? Anyway, I'd like to have a look at it. Thanks, eric ---------------------------------------------- eric jones 515 Congress Ave www.enthought.com Suite 1614 512 536-1057 Austin, Tx 78701 then the their
PYTHONPATH).
A version of sitecustomize.py is attached to this message and I'd like to propose adding this script to scipy directory. The attached sitecustomize.py script contains also a fix to distutils.util.get_platform function that is broken for Mac OSX system. This script might be useful also for back-porting other Python fixes/features to older Python versions that SciPy is supposed to support.
What do you think?
Pearu
On Sun, 29 Dec 2002, eric jones wrote:
Hey Pearu,
The script wasn't attached. Is this a mailing list issue, or just an oversight? Anyway, I'd like to have a look at it.
It wasn't oversight. I guess mailman stripped the attachment. Anyway, the script is inserted below. Additional comment on the script: OSDIR should be renamed to PYTHONPREFIX or MYPYTHONPREFIX or similar. Pearu #!/usr/bin/env python # # Purpose: # Facilitate installing/using Python extension modules to/from # non-standard locations when installation is carried out as # python setup.py install --prefix=$HOME # or # python setup.py install --prefix=$OSDIR # where the environment variable OSDIR contains arbitrary path. # # Recommended but optional environment (as defined in ~/.bashrc): # OSDIR=~/plat/`python -c 'import string;from distutils.util \ # import get_platform;print string.replace(get_platform()," ","_")'` # export OSDIR # export PYTHONPATH=.:$HOME/site-python # export PATH=$OSDIR/bin:$PATH # # Usage: # Copy sitecustomize.py to location (e.g. to $HOME/site-python/) # where Python can import it. # # Features: # * Add <prefix>/lib/python<version>/site-packages to sys.path # * Add <prefix>/include/python<version> to the list of include_dirs # of distutils.build_ext.build_ext class # * Fix distutils.util.get_platform bug on Mac OS X # # Author: Pearu Peterson, Dec 2002 # import os import sys import site import string ######## from distutils.sysconfig import get_python_inc l=len(sys.path) prefixes = [os.environ.get('OSDIR'),os.environ.get('HOME')] extra_include_dirs = {} for prefix in prefixes: if prefix: if os.sep == '/': sitedirs = [os.path.join(prefix, "lib", "python" + sys.version[:3], "site-packages"), os.path.join(prefix, "lib", "site-python")] else: sitedirs = [prefix, os.path.join(prefix, "lib", "site-packages")] for sitedir in sitedirs: if os.path.isdir(sitedir): site.addsitedir(sitedir) incdir = get_python_inc(prefix=prefix) if os.path.isdir(incdir): extra_include_dirs[incdir] = 1 extra_include_dirs = extra_include_dirs.keys() sys.path = sys.path[l:] + sys.path[:l] ########### from distutils.command import build_ext old_build_ext = build_ext.build_ext class site_build_ext(old_build_ext): def finalize_options (self): old_build_ext.finalize_options(self) l = len(self.include_dirs) - 1 for d in extra_include_dirs: try: self.include_dirs.index(d) except ValueError: # insert just before system include_dir self.include_dirs.insert(l,d) l = l - 1 build_ext.build_ext = site_build_ext ########### from distutils import util old_get_platform = util.get_platform def get_platform(): return string.replace(old_get_platform()," ","_") util.get_platform = get_platform ########### if __name__=='__main__': site._test()
participants (2)
-
eric jones -
Pearu Peterson