Upheaval in build commands: almost done!
Hi all -- as I mentioned last night, I've been hacking on the build_* commands lately, with the goal of putting all temporary compiler by-products under build/temp.<plat>, and allowing "in-place" building of extensions. Everything is done now, and for the most part It Works For Me (TM). (I just remembered that I haven't tested the new --inplace option on the build_ext command -- arg!) Windows support isn't there yet -- I have to hack on the MSVCCompiler class a bit, and I think I will take advantage of this to reduce overlap and increase uniformity between UnixCCompiler and MSVCCompiler -- ie. it'll take parallel printouts and a couple of hours, so I'll save it for tomorrow. Anyways, if you're a on a Unix-y platform, please check out the latest CVS and try building some distributions with extensions and/or C libraries. NumPy and PIL are as usual the canonical examples (setup script free with the Distutils!), but if you're feeling adventurous you might try distutilizing some other distribution. Oh yeah, the setup.py currently distributed with NumPy won't work with the CVS version of Distutils; you'll have to replace it with examples/numpy_setup.py. It was the renaming thing in command classes that did it, y'know. Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ Never underestimate the power of human stupidity.
I'm trying to use distutils (CVS from 3/6/2000) to build C++ extensions that use the Numeric library. I'm using Python 1.5.2, Redhat 6.1, Solaris 2.6 and Gcc 2.95.2 I'm pleased that I can build and install my extensions for both Solaris and Linux, now that I've built workarounds for the two problems I describe below. Given that distutils automatically builds libraries and objects in machine-dependent temporary directories, I find it much easier to use than the corresponding Makefiles. Here are two problems that should be addressed. Problem 1 - locating Python extension include files. Numeric has its own include files, which are installed under /usr[/local]/include/python1.5/Numeric I want to specify the location of the Numeric include files in an installation/machine independent way. I added the following into my setup.py file: ###################### from distutils import sysconfig #determine location of Python include files, to locate Numeric includes config_h = sysconfig.get_config_h_filename() py_inc = os.path.dirname(config_h) NUM_INC = os.path.join(py_inc, 'Numeric') ################################# I've attached a patch for sysconfig.py that defines get_python_inc(), to provide a cleaner way to find where the python includes are located. This allows a user to find the python include files as follows: py_inc = sysconfig.get_python_inc() NUM_INC = os.path.join(py_inc, 'Numeric') Problem 2 - specifying architecture/compiler specific libraries to setup.py When I link my extension on Linux (RH 6.1), I need to specify 'libraries': ['gcc', 'stdc++','pthread'], However, when I link my extensions on Solaris 2.6, these libraries do not exist. (At this point, we start approaching the complex world of GNU 'autoconf', that dynamically checks for the existence of specified libraries! ) Here's a solution, is there a better one? from distutils.util import get_platform plat = get_platform() # machine dependent C/C++ support libs if plat == 'linux-i586' : CLIBS=['gcc', 'stdc++','pthread'] else: CLIBS=[] . . . ext_modules = [ ('Perp.a1pp.pulsepairc', {'sources': ['a1pp/pulsepair.cc', 'a1pp/pulsepair_wrap.cc'], 'libraries': CLIBS, }, ), -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel@ucar.edu *** sysconfig.py.orig Tue Mar 7 08:44:18 2000 --- sysconfig.py Tue Mar 7 08:45:18 2000 *************** *** 25,30 **** --- 25,37 ---- return os.path.join(exec_prefix, "include", "python" + sys.version[:3], "config.h") + def get_python_inc(): + """Return full pathname of installed config.h file.""" + if os.name == "nt": + return os.path.join(exec_prefix, "include") + else: + return os.path.join(exec_prefix, + "include", "python" + sys.version[:3]) def get_makefile_filename(): """Return full pathname of installed Makefile from the Python build."""
On 07 March 2000, Joe Van Andel said:
I'm trying to use distutils (CVS from 3/6/2000) to build C++ extensions that use the Numeric library. I'm using Python 1.5.2, Redhat 6.1, Solaris 2.6 and Gcc 2.95.2
Great -- first successful build of C++ extensions, AFAIK. I hope the compiler interface is holding up under the strain. Planning to try it under Windows?
I'm pleased that I can build and install my extensions for both Solaris and Linux, now that I've built workarounds for the two problems I describe below. Given that distutils automatically builds libraries and objects in machine-dependent temporary directories, I find it much easier to use than the corresponding Makefiles.
Oh good, another completely untested feature getting a workout. The intention of having per-architecture build directories was for installers and/or developers to be able to do this: [on arch. A, eg. Linux] python setup.py build [on arch. B, eg. Solaris] python setup.py build and have *no* interference between the two builds, ie. everything would be in the build/temp.<plat> directory. You should be able to build on A, build on B, install on A, install on B. Whatever. I have not tested any of this for real, though, and it's only been in the source since the weekend: so, does it work for you?
Here are two problems that should be addressed.
Problem 1 - locating Python extension include files.
Hmm, yeah, I worried about this a bit when hacking Numeric's setup script to install header files. Didn't have an answer then.
I want to specify the location of the Numeric include files in an installation/machine independent way. I added the following into my setup.py file: [...] config_h = sysconfig.get_config_h_filename() py_inc = os.path.dirname(config_h) NUM_INC = os.path.join(py_inc, 'Numeric')
Nice kludge, but subtly wrong: the config.h directory is for platform-specific header files. You most likely want the non-platform-specific header file directory, ie. where Python.h and friends live. I think your patch (add 'get_python_inc()') is 95% exactly right. Here's my adaptation of it, with Mac support based on Corran Webster's '_init_mac()' (checked in last night -- thanks Corran!): def get_python_inc (plat_specific=0): """Return the directory containing installed Python header files. If 'plat_specific' is false (the default), this is the path to the non-platform-specific header files, i.e. Python.h and so on; otherwise, this is the path to platform-specific header files (namely config.h).""" the_prefix = (plat_specific and exec_prefix or prefix) if os.name == "posix": return os.path.join (the_prefix, "include", "python" + sys.version[:3]) elif os.name == "nt": return os.path.join (the_prefix, "Include") # include or Include? elif os.name == "mac": return os.path.join (the_prefix, "Include") else: raise DistutilsPlatformError, \ ("I don't know where Python installs its C header files " + "on platform '%s'") % os.name Ummm, should that be "Include" on Windows for consistency with "Lib"? (Yes yes, I know it doesn't strictly matter, but I'm a stickler for aesthetic consistency, even in a tty-style UI.) 'get_config_h_filename()' should change to use the new function... there, done. And come to think of it, why not add 'get_python_lib()'? The 'install_py' command should probably use this instead of some ugly GLOBAL exported by sysconfig, and it could be handy for setup scripts that want to do naughty things but still play nice (ie. use portable directory names). Here's 'get_python_lib()': def get_python_lib (plat_specific=0, standard_lib=0): """Return the directory containing the Python library (standard or site additions). If 'plat_specific' is true, return the directory containing platform-specific modules, i.e. any module from a non-pure-Python module distribution; otherwise, return the platform-shared library directory. If 'standard_lib' is true, return the directory containing standard Python library modules; otherwise, return the directory for site-specific modules.""" the_prefix = (plat_specific and exec_prefix or prefix) if os.name == "posix": libpython = os.path.join (the_prefix, "lib", "python" + sys.version[:3]) if standard_lib: return libpython else: return os.path.join (libpython, "site-packages") elif os.name == "nt": if standard_lib: return os.path.join (the_prefix, "Lib") else: return the_prefix elif os.name == "mac": if platform_specific: if standard_lib: return os.path.join (exec_prefix, "Mac", "Plugins") else: raise DistutilsPlatformError, \ "OK, where DO site-specific extensions go on the Mac?" else: if standard_lib: return os.path.join (prefix, "Lib") else: raise DistutilsPlatformError, \ "OK, where DO site-specific modules go on the Mac?" else: raise DistutilsPlatformError, \ ("I don't know where Python installs its library " + "on platform '%s'") % os.name # get_python_lib () Hint hint: I need help with the Mac support!
Problem 2 - specifying architecture/compiler specific libraries to setup.py
When I link my extension on Linux (RH 6.1), I need to specify 'libraries': ['gcc', 'stdc++','pthread'],
Nope, I still don't have an answer to this one.
(At this point, we start approaching the complex world of GNU 'autoconf', that dynamically checks for the existence of specified libraries! )
Well, except for this non-answer. If I start rewriting Autoconf in Python for Distutils 1.0, though, Guido will kill me (there's already too damn much code in this thing!).
Here's a solution, is there a better one?
from distutils.util import get_platform plat = get_platform()
# machine dependent C/C++ support libs if plat == 'linux-i586' : CLIBS=['gcc', 'stdc++','pthread'] else: CLIBS=[]
That would be the "everybody-has-a-personal-imake-database-in-their- setup-script" non-solution. Ick. See also the pil_setup.py example in the distutils source, which goes for the "everyone-knows-what-libraries- they-have-installed-and-edits-the-setup-script" non-solution, which is approximately as nasty. So far, I have yet to be convinced that there is *any* long-term solution short of rewriting Autoconf in Python. (Distutils 2.0, anyone?) And oh yeah, this has to be an Autoconf that runs on Unix, Windows, and Mac OS. Good thing there's a portable C/C++ compiler framework! Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ God is real, unless declared integer.
I'm new on this list, so please forgive me if I'm treading on familiar ground, but I didn't see this issue raised when I browsed through the last month or so of archives. Distutils for Windows NT,9x,2K seems to have a big hole in the way it forms command-lines for the compiler and friends: it doesn't account for the fact that filenames and pathnames may have spaces in them and thus may need to be quoted on the command-line passed to external tools (compiler, linker). Given that the standard Python 1.5.2 distribution installs itself on "C:\Program Files\Python" this leads to potentially problematic command-lines of the form cl -IC:\Program Files\Python\include link C:\Program Files\Python\Libs\python15.lib Both of these command lines will fail, because cl can't find the directory "C:\Program" and can't find "Files\Python\include.obj" and link can't find "C:\Program" or "Files\Python\Libs\python15.lib". CL and LINK both reparse the command-line so even if "-IC:/Program Files/Python/include" is passed as a single argument to os.spawnv(), it is split into two by cl's command-line parser: Python 1.5.2 (#0, Feb 2 2000, 22:07:42) [MSC 32 bit (Intel)] on win32 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> from distutils.spawn import * >>> spawn(['cl','-c','-IC:/Program Files/Python/include','client.cpp']) Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. Command line warning D4024 : unrecognized source file type 'Files/Python/include', object file assumed Command line warning D4027 : source file 'Files/Python/include' ignored client.cpp client.cpp(7) : fatal error C1083: Cannot open include file: 'python.h': No such file or directory Traceback (innermost last): File "<stdin>", line 1, in ? File "C:\Program Files\python\python-1.5.2\distutils\spawn.py", line 37, in spawn _spawn_nt (cmd, search_path, verbose, dry_run) File "C:\Program Files\python\python-1.5.2\distutils\spawn.py", line 74, in _spawn_nt raise DistutilsExecError, \ distutils.errors.DistutilsExecError: command 'cl' failed with exit status 2 >>> The command-lines should be: cl -I"C:\Program Files\Python\include" link "C:\Program Files\Python\Libs\python15.lib" which work: Python 1.5.2 (#0, Feb 2 2000, 22:07:42) [MSC 32 bit (Intel)] on win32 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> from distutils.spawn import * >>> spawn(['cl','-c','-I"C:/Program Files/Python/include"','client.cpp']) Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. client.cpp >>> I started hacking at the msvccompiler.py source to put quotes around filenames passed as arguments, but then realized that it would probably be better to implement a "wrapfilename(fname)" method in the ccompiler class and then subclass it appropriately for *ix (return fname) and WinNT (return '"%s"' % fname). Then every time a filename or path gets added to a command-line it must be processed with wrapfilename(). For instance: inputOpt = fileOpt + wrapfilename(srcFile) Does this sound like a reasonable approach? Am I missing a boat somewhere? Implementing this looks as thought it will require modifying (slightly) a number functions in ccompiler.py and msvccompiler.py, so I would like feedback from all of you more experienced folks on this list before I spend any time at it. I also might not be the best person to try, being a greenhorn and thus less prone to catch where such a modification would break existing code, but I am willing to make a pass at it if others don't have the time. Alternately, one could modify _spawn_nt to wrap each of its arguments in double-quotes, but that seems potentially more problematic (Some programs may not transparently discard quotation marks on command-line arguments. Consider the behavior of the find command.). Thanks for listening, Jonathan Gilligan
Greg Ward writes:
Ummm, should that be "Include" on Windows for consistency with "Lib"? (Yes yes, I know it doesn't strictly matter, but I'm a stickler for aesthetic consistency, even in a tty-style UI.)
The installer created the directory "include" on my box. Inconsistent with "Lib", but that's deployment for you! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
On 09 March 2000, Jonathan M. Gilligan said:
I'm new on this list, so please forgive me if I'm treading on familiar ground, but I didn't see this issue raised when I browsed through the last month or so of archives.
Distutils for Windows NT,9x,2K seems to have a big hole in the way it forms command-lines for the compiler and friends: it doesn't account for the fact that filenames and pathnames may have spaces in them and thus may need to be quoted on the command-line passed to external tools (compiler, linker).
Yeah, that did come up not *too* long ago. It was during a big flurry of activity on the Windows support, so no surprise that you missed it. Bottom line: Thomas Heller submitted a patch several weeks ago, and I *finally* got around to applying it this weekend. It's in the CVS version.
I started hacking at the msvccompiler.py source to put quotes around filenames passed as arguments, but then realized that it would probably be better to implement a "wrapfilename(fname)" method in the ccompiler class and then subclass it appropriately for *ix (return fname) and WinNT (return '"%s"' % fname). Then every time a filename or path gets added to a command-line it must be processed with wrapfilename(). For instance:
inputOpt = fileOpt + wrapfilename(srcFile)
Does this sound like a reasonable approach? Am I missing a boat somewhere?
But what about arguments with spaces in them that *aren't* filenames? (And what about spawning programs on Windows *other* than the compiler/linker? Granted that's not happening right now in Distutils, but who knows what'll happen in the future?)
Alternately, one could modify _spawn_nt to wrap each of its arguments in double-quotes, but that seems potentially more problematic (Some programs may not transparently discard quotation marks on command-line arguments. Consider the behavior of the find command.).
That's exactly what we ended up doing. Your point about some programs not dealing with quote characters is a bit unsettling, though. *sigh* "Those who do not learn from Unix are doomed to reinvent it -- badly." Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ Earn cash in your spare time -- blackmail your friends!
participants (4)
-
Fred L. Drake, Jr. -
Greg Ward -
Joe Van Andel -
Jonathan M. Gilligan