Greg Ward wrote:
On 06 July 2000, Rene Liebscher said:
I didn't changed bcppcompiler.py because I can't test it.
OK. Lyle, are you out there and following this? I can try to hack up bcppcompiler.py to agree with msvccompiler.py and cygwincompiler.py, but I can't test it either. But I can test it now. I found the problem Lyle had in it. (He forgot to close the file in which he wrote the export symbols, and the compiler didn't want use an open file.) I also added some comments and a extension concerning libraries. It now looks first if there is a library called bcpp_{libname}.lib before it tries {libname}.lib. So you can have both,
the default msvc libraries and special libraries for Borland C++. (They are not compatible.) This is useful if you have installed your Python in a network and user use different compilers. (Or for someone like me, who has now six different compilers on one machine to test distutils compiler classes.)
I vote we drop 'export_symbol_file' from Extension -- ie. require that the symbols to export be listed in a Python list in your setup script, rather than in an external file -- and see if anyone howls. It is your decision, export_symbol_file is now no longer used anywhere, but it still exists as parameter to Extension.
Detailed comments on the patch...
--- distutils.orig/distutils/cygwinccompiler.py Fri Jun 30 17:17:55 2000 +++ dp/distutils/cygwinccompiler.py Wed Jul 5 12:29:57 2000 @@ -6,11 +6,34 @@ cygwin in no-cygwin mode). """
+# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a
Ouch! So the Cygwin support is definitely not ready for prime time with 1.5.2. My understanding is that #1 has been fixed in the current Python 2.0 CVS code. Not yet, but there is a patch in SourceForges open patches list.
What about #2? Should we try to convince Guido to include libpython20.a the 2.0 distribution to save Cygwin/Mingw32 users the bother of generating it?
Or maybe "build_ext" can grow a little bit of compiler-sensitivity for Cygwin/Mingw32 and run the above command to generate libpython20.a the first time the user tries to build an extension with Cygwin/Mingw32. Come to think of it, this solves the problem nicely for 1.5.2. And hey, we could even do the same thing for config.h, but that's rather more dangerous -- modifying an existing file rather than generating a new one. Worth considering, though. I think he should add a def-file for python20.dll in the distribution. Every serious compiler can build from this an import library. (Many can also use the dll to build it, but most fail because Python exports not only functions, it also exports data.) (Borland can also translate MSVC import libraries into its own format.)
We could also add a tools directory or something similar. In this directory we could put some python programs which generate an import library for a given def-file, library or dll. They could called ????_def2lib.py, ????_lib2lib.py and ????_dll2lib.py, where ???? stands for the compiler (bcpp,...). This would help users to use other dlls or libraries they want to use for extension. This would also the place to put a replacement for the python 1.5.2 config.h file. I think the python 2.0 config.h should work fine, after changing references from 2.0 to 1.5 (#pragma comment(lib,"python20.lib")
+# * cygwin gcc 2.91 works (after patching some include files) +# * mingw32 gcc 2.95.2 works (it doesn't support gcc -shared) +# * cygwin gcc 2.95.2 doesn't work now +# - its dllwrap doesn't work at my machine +# - target had to be changed to something like i686-cygwin32 +# - mingw32-mode is specified by -mno-cygwin +# - using gcc -shared instead dllwrap didn't work because some problems with +# the import library, maybe it needs some additional parameters
AAUUGGHH!!! (What more can I say?)
- # XXX the result of the check should be returned! + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK
Ick. Magic numbers bad. Strings good. Intelligent exception objects (often) better. Think about it. Not magic numbers, more a kind of level, higher number mean a greater
gcc 2.91 is no problem for us. People should use a newer compiler. But the differences betweeen these two gcc 2.9.5 are a problem, mingw32 doesn't support gcc -shared yet, and cygwin's dllwrap seems to have a bug (It fails to reallocate 1.7 GByte of Memory.) Cygwin's gcc -shared wants other symbols than dlltool creates in the import library. Is there someone out there who has already successful created a extension module with the new cygwin version? If yes, please send me the commandlines you used for this. probability that compiling will successful.
Is it necessary to make a .def file if there are multiple export symbols? Or can they go on the command line? (I think I'd rather put them on the command-line: more explicit, and one less temp file to worry about.)
Normally gcc exports all symbols if you don't specify any exports related options. But you can't rely on it, I also found some extensions build without exports. This why we have to specify them. There is an exclude-symbols option, but I never found a export-symbols or include-symbols or something similar. It seems we have to build this def-file.
- self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols
I understand the reason for this change, but we need to make sure there isn't code out there that relies on export_symbols always being a list (ie. never None). It was never used by build_ext and in the compiler interface it has a default value of None. So it is unlikely someone will even take notice of it.
And now something new. Is anyone out there using the LCC-Win32 compiler? (http://www.cs.virginia.edu/~lcc-win32/) It is a small C compiler, but it comes with a IDE (gcc and free BCC55 don't) and it is able to build python extensions. I have a compiler class for it, which is already able to build extensions. So if anyone is interested or maybe want it to complete, I will post it to the mailing list. kind regards Rene Liebscher diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/bcppcompiler.py distutils.patched/distutils/bcppcompiler.py --- distutils.orig/distutils/bcppcompiler.py Wed Jun 28 03:20:35 2000 +++ distutils.patched/distutils/bcppcompiler.py Wed Jul 12 14:33:34 2000 @@ -11,17 +11,6 @@ # someone should sit down and factor out the common code as # WindowsCCompiler! --GPW -# XXX Lyle reports that this doesn't quite work yet: -# """...but this is what I've got so far. The compile step works fine but -# when it runs the link step I get an "out of memory" failure. Since -# spawn() echoes the command it's trying to spawn, I can type the link line -# verbatim at the DOS prompt and it links the Windows DLL correctly -- so -# the syntax is correct. There's just some weird interaction going on when -# it tries to "spawn" the link process from within the setup.py script. I'm -# not really sure how to debug this one right off-hand; obviously there's -# nothing wrong with the "spawn()" function since it's working properly for -# the compile stage.""" - __revision__ = "$Id: bcppcompiler.py,v 1.1 2000/06/28 01:20:35 gward Exp $" @@ -123,6 +112,9 @@ elif ext in self._cpp_extensions: input_opt = "-P" + src = os.path.normpath(src) + obj = os.path.normpath(obj) + output_opt = "-o" + obj self.mkpath (os.path.dirname (obj)) @@ -230,24 +222,55 @@ else: ldflags = self.ldflags_shared + # Borland C++ has problems with '/' in paths + objects = map(os.path.normpath,objects) + startup_obj = 'c0d32' - libraries.append ('mypylib') + # either exchange python15.lib in the python libs directory against + # a Borland-like one, or create one with name bcpp_python15.lib + # there and remove the pragmas from config.h + #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt') # Create a temporary exports file for use by the linker head, tail = os.path.split (output_filename) modname, ext = os.path.splitext (tail) - def_file = os.path.join (build_temp, '%s.def' % modname) + temp_dir = os.path.dirname(objects[0]) # preserve tree structure + def_file = os.path.join (temp_dir, '%s.def' % modname) f = open (def_file, 'w') f.write ('EXPORTS\n') for sym in (export_symbols or []): f.write (' %s=_%s\n' % (sym, sym)) - - ld_args = ldflags + [startup_obj] + objects + \ - [',%s,,' % output_filename] + \ - libraries + [',' + def_file] + f.close() + + # start building command line + # flags and options + ld_args = list(ldflags) # don't use a reference, use a separate copy + for l in library_dirs: + ld_args.append("/L%s" % os.path.normpath(l)) + # objects list + ld_args.extend([startup_obj]) + ld_args.extend(objects) + # name of dll file + ld_args.extend([',',output_filename]) + # no map file + ld_args.extend([',']) + # start libraries + ld_args.extend([',']) + for lib in libraries: + # see if we find it and if there is a bcpp specific lib (bcpp_xxx.lib) + l = self.find_library_file(library_dirs,lib,debug) + if l == None: + ld_args.append(lib) + # probably an bcpp internal library + # self.warn('library %s not found.' % lib) + else: + # full name which prefers bcpp_xxx.lib over xxx.lib + ld_args.append(l) + # def file for export symbols + ld_args.extend([',',def_file]) if extra_preargs: ld_args[:0] = extra_preargs @@ -325,15 +348,28 @@ def runtime_library_dir_option (self, dir): raise DistutilsPlatformError, \ - "don't know how to set runtime library search path for MSVC++" + "don't know how to set runtime library search path for Borland C++" def library_option (self, lib): return self.library_filename (lib) - def find_library_file (self, dirs, lib): - + def find_library_file (self, dirs, lib, debug=0): + # find library file + # bcpp_xxx.lib is better than xxx.lib + # and xxx_d.lib is better than xxx.lib if debug is set for dir in dirs: + if debug: + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib + "_d")) + if os.path.exists (libfile): + return libfile + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib)) + if os.path.exists (libfile): + return libfile + if debug: + libfile = os.path.join (dir, self.library_filename (lib + '_d')) + if os.path.exists (libfile): + return libfile libfile = os.path.join (dir, self.library_filename (lib)) if os.path.exists (libfile): return libfile diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py distutils.patched/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Fri Jul 7 22:41:21 2000 +++ distutils.patched/distutils/command/build_ext.py Wed Jul 12 13:57:48 2000 @@ -283,6 +283,9 @@ # Medium-easy stuff: same syntax/semantics, different names. ext.runtime_library_dirs = build_info.get('rpath') ext.export_symbol_file = build_info.get('def_file') + if ext.export_symbol_file is not None: + self.warn("export_symbol_file is no longer supported, " + "comments and questions to <distutils-sig@python.org>") # Non-trivial stuff: 'macros' split into 'define_macros' # and 'undef_macros'. @@ -420,16 +423,14 @@ objects.extend (ext.extra_objects) extra_args = ext.extra_link_args or [] - # Bunch of fixing-up we have to do for Microsoft's linker. - if self.compiler.compiler_type == 'msvc': - self.msvc_prelink_hack(sources, ext, extra_args) self.compiler.link_shared_object ( objects, ext_filename, - libraries=ext.libraries, + libraries=self.get_libraries(ext), library_dirs=ext.library_dirs, runtime_library_dirs=ext.runtime_library_dirs, extra_postargs=extra_args, + export_symbols=self.get_export_symbols(ext), debug=self.debug, build_temp=self.build_temp) @@ -511,44 +512,6 @@ # find_swig () - - # -- Hooks 'n hacks ------------------------------------------------ - - def msvc_prelink_hack (self, sources, ext, extra_args): - - # XXX this is a kludge! Knowledge of specific compilers or - # platforms really doesn't belong here; in an ideal world, the - # CCompiler interface would provide access to everything in a - # compiler/linker system needs to build Python extensions, and - # we would just do everything nicely and cleanly through that - # interface. However, this is a not an ideal world and the - # CCompiler interface doesn't handle absolutely everything. - # Thus, kludges like this slip in occasionally. (This is no - # excuse for committing more platform- and compiler-specific - # kludges; they are to be avoided if possible!) - - def_file = ext.export_symbol_file - - if def_file is not None: - extra_args.append ('/DEF:' + def_file) - else: - modname = string.split (ext.name, '.')[-1] - extra_args.append('/export:init%s' % modname) - - # The MSVC linker generates .lib and .exp files, which cannot be - # suppressed by any linker switches. The .lib files may even be - # needed! Make sure they are generated in the temporary build - # directory. Since they have different names for debug and release - # builds, they can go into the same directory. - implib_file = os.path.join ( - self.implib_dir, - self.get_ext_libname (ext.name)) - extra_args.append ('/IMPLIB:' + implib_file) - self.mkpath (os.path.dirname (implib_file)) - - # msvc_prelink_hack () - - # -- Name generators ----------------------------------------------- # (extension names, filenames, whatever) @@ -578,5 +541,33 @@ if os.name == 'nt' and self.debug: return apply (os.path.join, ext_path) + '_d.lib' return apply (os.path.join, ext_path) + '.lib' + + + def get_export_symbols (self, ext): + # For all platforms and compiler doesn't export all symbols + # by default, we have to specify the module's init function + # if the user doesn't provide any export information. + # All other platforms and compilers should ignore such + # export information. + if ext.export_symbols is None: + # if no exports are specified we can't build a working + # python module. + # so let us export the module's init function + return ["init" + string.split(ext.name,'.')[-1]] + else: + return ext.export_symbols + + def get_libraries (self, ext): + # The python library is always needed on Windows. + # We need the python version without the dot, eg. '15' + # MSVC doesn't really need this, because the pragma in config.h + if sys.platform == "win32": + pythonlib = ("python%d%d" % + (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) + # don't extend ext.libraries, it may be shared with other + # extensions, it is a reference to the original list + return ext.libraries + [pythonlib] + else: + return ext.libraries # class build_ext diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/cygwinccompiler.py distutils.patched/distutils/cygwinccompiler.py --- distutils.orig/distutils/cygwinccompiler.py Fri Jun 30 00:57:55 2000 +++ distutils.patched/distutils/cygwinccompiler.py Wed Jul 12 14:17:57 2000 @@ -10,6 +10,29 @@ __revision__ = "$Id: cygwinccompiler.py,v 1.2 2000/06/29 22:57:55 gward Exp $" +# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a +# +# see also http://starship.python.net/crew/kernr/mingw32 +# +# * cygwin gcc 2.91 works (after patching some include files) +# * mingw32 gcc 2.95.2 works (it doesn't support gcc -shared) +# * cygwin gcc 2.95.2 doesn't work now +# - its dllwrap doesn't work at my machine +# - using gcc -shared instead dllwrap didn't work because some problems with +# the import library, maybe it needs some additional parameters. +# It tries to import _imp__XXX but dlltool creates an import library +# with __imp__XXX symbols. + + + import os,sys,string from distutils import sysconfig from distutils.unixccompiler import UnixCCompiler @@ -20,34 +43,41 @@ def check_config_h(): """Checks if the GCC compiler is mentioned in config.h. If it is not, - compiling probably doesn't work, so print a warning to stderr. + compiling probably doesn't work. """ - - # XXX the result of the check should be returned! + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK from distutils import sysconfig import string,sys + # if sys.version contains GCC then python was compiled with + # GCC, and the config.h file should be OK + if -1 == string.find(sys.version,"GCC"): + pass # go to the next test + else: + return 2 + try: # It would probably better to read single lines to search. # But we do this only once, and it is fast enough f=open(sysconfig.get_config_h_filename()) s=f.read() f.close() - try: - # is somewhere a #ifdef __GNUC__ or something similar - string.index(s,"__GNUC__") - except ValueError: - sys.stderr.write ("warning: "+ - "Python's config.h doesn't seem to support your compiler.\n") + + # is somewhere a #ifdef __GNUC__ or something similar + if -1 == string.find(s,"__GNUC__"): + return -1 + else: + return 1 except IOError: # if we can't read this file, we cannot say it is wrong # the compiler will complain later about this file as missing pass - - -# This is called when the module is imported, so we make this check only once -# XXX why not make it only when the compiler is needed? -check_config_h() + return 0 class CygwinCCompiler (UnixCCompiler): @@ -59,14 +89,20 @@ dry_run=0, force=0): + UnixCCompiler.__init__ (self, verbose, dry_run, force) + if check_config_h()<=0: + self.warn( + "Python's config.h doesn't seem to support your compiler. " + "Compiling may fail because of undefined preprocessor macros.") + # Hard-code GCC because that's what this is all about. # XXX optimization, warnings etc. should be customizable. self.set_executables(compiler='gcc -O -Wall', - compiler_so='gcc -O -Wall', + compiler_so='gcc -O -Wall -mcygwin', linker_exe='gcc', - linker_so='dllwrap --target=i386-cygwin32') + linker_so='dllwrap -mcygwin') # cygwin and mingw32 need different sets of libraries self.dll_libraries=[ @@ -96,53 +132,49 @@ if libraries == None: libraries = [] - # Additional libraries: the python library is always needed on - # Windows we need the python version without the dot, eg. '15' - - pythonlib = ("python%d%d" % - (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) - libraries.append(pythonlib) - libraries.extend(self.dll_libraries) - - # name of extension + # Additional libraries + # libraries.extend(...) would affect the list we got as parameter + # and this can come from anywhere + libraries = libraries + self.dll_libraries + + # we want to put some files in the same directory as the + # object files are, build_temp doesn´t help much - # XXX WRONG WRONG WRONG - # this is NOT the place to make guesses about Python namespaces; - # that MUST be done in build_ext.py + # name of dll to give the helper files (def, lib, exp) the same name + dll_name,dll_extension = os.path.splitext(os.path.basename(output_filename)) - if not debug: - ext_name = os.path.basename(output_filename)[:-len(".pyd")] - else: - ext_name = os.path.basename(output_filename)[:-len("_d.pyd")] + # where are the object files + temp_dir = os.path.dirname(objects[0]) - def_file = os.path.join(build_temp, ext_name + ".def") - #exp_file = os.path.join(build_temp, ext_name + ".exp") - #lib_file = os.path.join(build_temp, 'lib' + ext_name + ".a") + # generate the filenames for these files + def_file = None # this will be done later, if necessary + exp_file = os.path.join(temp_dir, dll_name + ".exp") + lib_file = os.path.join(temp_dir, 'lib' + dll_name + ".a") - # Make .def file - # (It would probably better to check if we really need this, - # but for this we had to insert some unchanged parts of - # UnixCCompiler, and this is not what we want.) - f = open(def_file,"w") - f.write("EXPORTS\n") # intro - if export_symbols == None: - # export a function "init" + ext_name - f.write("init" + ext_name + "\n") - else: - # if there are more symbols to export write them into f + # check what we got in export_symbols + if export_symbols is not None: + # Make .def file + # (It would probably better to check if we really need this, + # but for this we had to insert some unchanged parts of + # UnixCCompiler, and this is not what we want.) + def_file = os.path.join(temp_dir, dll_name + ".def") + f = open(def_file,"w") + f.write("EXPORTS\n") # intro for sym in export_symbols: f.write(sym+"\n") - f.close() - - if extra_preargs == None: + f.close() + + + if extra_preargs is None: extra_preargs = [] extra_preargs = extra_preargs + [ #"--verbose", #"--output-exp",exp_file, #"--output-lib",lib_file, - "--def",def_file ] + if def_file: + extra_preargs.extend(["--def",def_file]) # who wants symbols and a many times larger output file # should explicitely switch the debug mode on @@ -150,7 +182,7 @@ # (On my machine unstripped_file = stripped_file + 254KB # 10KB < stripped_file < ??100KB ) if not debug: - extra_preargs = extra_preargs + ["-s"] + extra_preargs.append("-s") UnixCCompiler.link_shared_object(self, objects, @@ -185,15 +217,13 @@ self.set_executables(compiler='gcc -mno-cygwin -O -Wall', compiler_so='gcc -mno-cygwin -O -Wall', linker_exe='gcc -mno-cygwin', - linker_so='dllwrap' - + ' --target=i386-mingw32' - + ' --entry _DllMain@12') - # mingw32 doesn't really need 'target' and cygwin too (it seems, - # it is enough to specify a different entry point) + linker_so='dllwrap -mno-cygwin -mthreads') + #+ ' --entry _DllMain@12') + # mingw32-mode on cygwin needs to specify a different entry point # no additional libraries need # (only msvcrt, which is already added by CygwinCCompiler) - # __init__ () - + # __init__ () + # class Mingw32CCompiler diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/extension.py distutils.patched/distutils/extension.py --- distutils.orig/distutils/extension.py Sat Jun 24 02:18:24 2000 +++ distutils.patched/distutils/extension.py Wed Jul 12 14:01:03 2000 @@ -111,7 +111,7 @@ self.extra_objects = extra_objects or [] self.extra_compile_args = extra_compile_args or [] self.extra_link_args = extra_link_args or [] - self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols self.export_symbol_file = export_symbol_file # class Extension diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/msvccompiler.py distutils.patched/distutils/msvccompiler.py --- distutils.orig/distutils/msvccompiler.py Fri Jun 30 21:37:59 2000 +++ distutils.patched/distutils/msvccompiler.py Wed Jul 12 14:01:03 2000 @@ -380,6 +380,17 @@ ld_args = (ldflags + lib_opts + export_opts + objects + ['/OUT:' + output_filename]) + # The MSVC linker generates .lib and .exp files, which cannot be + # suppressed by any linker switches. The .lib files may even be + # needed! Make sure they are generated in the temporary build + # directory. Since they have different names for debug and release + # builds, they can go into the same directory. + dll_name, dll_ext = os.path.splitext(os.path.basename(output_filename)) + implib_file = os.path.join ( + os.path.dirname(objects[0]), + self.static_lib_format % (dll_name, self.static_lib_extension)) + ld_args.append ('/IMPLIB:' + implib_file) + if extra_preargs: ld_args[:0] = extra_preargs if extra_postargs: