---- CygwinCCompiler ------------ OK, it should now be better to read. I also included your other changes (set_executables, build_temp.) ----- msvc hack ----------- I also had a look at this msvc hack in build_ext. I think it is now possible to get rid of it. It does several things which could be done in an other way or shouldn't be done at all. First, it tries to find a def-file in the source directory if none is given. I think one should specify this explicitely, who knows if it is really the right file and moreover it overwrites any export_symbols given to Extension(). Then it uses the module_name to create an export directive for the procedure 'init{modulename}'. It is possible to extract this name from the output filename (see (bcpp|cygwin)compiler.py ) I think this export parameter should be only used if neither a def file nor a list of symbols is given. (Then you could use your compiler classes also for non python dll's by specifying [] as export_symbols list.) And finally it uses the build_temp path for it implib directive. We have now this new extra parameter to link_shared_library, so this is also not a problem. When I looked at the code of build_ext I found you are not using export_symbols as parameter to link_shared_object. Also you are not using any of this export_* values from the Extension class. This should be changed. But there is one point which is not clear. There is no parameter at link_shared_object for export_symbols_file. Either we add this or we change the semantics of the existing export_symbols a bit. * export_symbols == None : no export symbols given => add init{module} * type(export_symbols) == ListType : export all symbols from this list * type(export_symbols) == StringType : take this as a filename of a def file ------ bcppcompiler ------------- Which version of Borland C you need to use it? I tried BC5. It doesn't have a linker named ilink32 (?) and it doesn't accept some of your parameters /q for bcc32 and /Gn /q for the linker (tlink32). Also it is using mypylib, which probably the stub library for python15.dll . Is this library somewhere to download or how can I create it? My BC version doesn't include files if they are in the source directory ( src/foo.c includes "foo.h" which is really src/foo.h ) so I had to specify this as an include_dir (-Isrc). If all versions of BC do the same, the directories of the source files should append to include_dir in the bcppcompiler class. Kind regards Rene Liebscher """distutils.cygwinccompiler Contains the CygwinCCompiler class, a subclass of UnixCCompiler that handles the Gnu Win32 C compiler. It also contains the Mingw32CCompiler class which handles the mingw32 compiler (same as cygwin in no-cygwin mode.) """ # created 2000/05/05, Rene Liebscher __revision__ = "$Id: cygwinccompiler.py,v 1.1 2000/06/21 03:33:03 gward Exp $" import os,sys,string from distutils import sysconfig from distutils.unixccompiler import UnixCCompiler # Because these compilers aren't configured in Python's config.h file by default # we should at least warn the user if he is using a unmodified version. def check_config_h(): """ checks, if the gcc-compiler is mentioned in config.h if it is not, compiling probably doesn't work """ from distutils import sysconfig import string,sys try: # It would probably better to read single lines to search. # But we do this only once, and it is fast enough f=open(sysconfig.get_config_h_filename()) s=f.read() f.close() try: # is somewhere a #ifdef __GNUC__ or something similar string.index(s,"__GNUC__") except ValueError: sys.stderr.write ("warning: "+ "Python's config.h doesn't seem to support your compiler.\n") except IOError: # if we can't read this file, we cannot say it is wrong # the compiler will complain later about this file as missing pass # This is called when the module is imported, so we make this check only once check_config_h() # XXX Things not currently handled: # * see UnixCCompiler class CygwinCCompiler (UnixCCompiler): compiler_type = 'cygwin' def __init__ (self, verbose=0, dry_run=0, force=0): UnixCCompiler.__init__ (self, verbose, dry_run, force) # our compiler uses other names # dllwrap: specification of entry point is not necessary self.set_executables(compiler='gcc -O -Wall', compiler_so='gcc -O -Wall', linker_exe='gcc', linker_so='dllwrap --target=i386-cygwin32') # cygwin and mingw32 need different sets of libraries self.dll_libraries=[ # cygwin shouldn't need msvcrt, # but without the dll's will crash # ( gcc version 2.91.57 ) # perhaps something about initialization # mingw32 needs it in all cases "msvcrt" ] # __init__ () def link_shared_object (self, objects, output_filename, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None): if libraries == None: libraries = [] # additional libraries # the python library is always needed on Windows # we need the python version without the dot, eg. '15' libraries = libraries + [ "python%d%d" % ( sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff) ] + self.dll_libraries # name of extension if not debug: ext_name = os.path.basename(output_filename)[:-len(".pyd")] else: ext_name = os.path.basename(output_filename)[:-len("_d.pyd")] def_file = os.path.join(build_temp, ext_name + ".def") #exp_file = os.path.join(build_temp, ext_name + ".exp") #lib_file = os.path.join(build_temp, 'lib' + ext_name + ".a") # Make .def file # (It would probably better to check if we really need this, # but for this we had to insert some unchanged parts of # UnixCCompiler, and this is not what we want.) f = open(def_file,"w") f.write("EXPORTS\n") # intro if export_symbols == None: # export a function "init" + ext_name f.write("init" + ext_name + "\n") else: # if there are more symbols to export write them into f for sym in export_symbols: f.write(sym+"\n") f.close() if extra_preargs == None: extra_preargs = [] extra_preargs = extra_preargs + [ #"--verbose", #"--output-exp",exp_file, #"--output-lib",lib_file, "--def",def_file ] # who wants symbols and a many times larger output file # should explicitely switch the debug mode on # otherwise we let dllwrap strip the output file # (On my machine unstripped_file = stripped_file + 254KB # 10KB < stripped_file < ??100KB ) if not debug: extra_preargs = extra_preargs + ["-s"] UnixCCompiler.link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, None, # export_symbols, we do this with our def-file debug, extra_preargs, extra_postargs, build_temp) # link_shared_object () # class CygwinCCompiler # the same as cygwin plus some additional parameters class Mingw32CCompiler (CygwinCCompiler): compiler_type = 'mingw32' def __init__ (self, verbose=0, dry_run=0, force=0): CygwinCCompiler.__init__ (self, verbose, dry_run, force) self.set_executables(compiler='gcc -mno-cygwin -O -Wall', compiler_so='gcc -mno-cygwin -O -Wall', linker_exe='gcc -mno-cygwin', linker_so='dllwrap' + ' --target=i386-mingw32' + ' --entry _DllMain@12') # mingw32 doesn't really need 'target' and cygwin too (it seems, # it is enough to specify a different entry point) # no additional libraries need # (only msvcrt, which is already added by CygwinCCompiler) # __init__ () # class Mingw32CCompiler
Rene Liebscher writes:
---- CygwinCCompiler ------------
OK, it should now be better to read. I also included your other changes (set_executables, build_temp.)
I just noticed that cygwinccompiler has been added to distutils :-)) One minor nitpick from cygwinccompiler.py # Because these compilers aren't configured in Python's config.h file by default # we should at least warn the user if he is using a unmodified version. def check_config_h(): """ checks, if the gcc-compiler is mentioned in config.h if it is not, compiling probably doesn't work """ FWIW I compile Python with Cygwin routinely and I do not have "__GNUC__" anywhere in my config.h file however given NHV:/f/jnk2> python Python 1.6a2 (#12, Jun 22 2000, 08:18:57) [GCC 2.95.2 19991024 (release-2)] on cygwin_98-4.101 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam Copyright 1995-2000 Corporation for National Research Initiatives (CNRI)
import sys sys.version '1.6a2 (#12, Jun 22 2000, 08:18:57) [GCC 2.95.2 19991024 (release-2)]'
perhaps checking for "GCC" in sys.version will do what we want :-)) Norman Vine
On 29 June 2000, Norman Vine said:
FWIW I compile Python with Cygwin routinely and I do not have "__GNUC__" anywhere in my config.h file [...]
sys.version '1.6a2 (#12, Jun 22 2000, 08:18:57) [GCC 2.95.2 19991024 (release-2)]'
perhaps checking for "GCC" in sys.version will do what we want :-))
No; as I understand it, the CygwinCCompiler class will allow you to build extensions using GCC (Cygwin or MingW32!) with the stock Python binary, as long as you have a libpython.a around. (It also requires changes to Python's config.h, but I believe Fred checked those in a few days ago.) This is a big win: let Guido's employer shell out for MSVC++, but you can use a free compiler to build extensions for Windows without having to build Python yourself. Greg -- Greg Ward - geek-on-the-loose gward@python.net http://starship.python.net/~gward/ A closed mouth gathers no foot.
Greg Ward wrote:
On 29 June 2000, Norman Vine said:
FWIW I compile Python with Cygwin routinely and I do not have "__GNUC__" anywhere in my config.h file [...]
sys.version '1.6a2 (#12, Jun 22 2000, 08:18:57) [GCC 2.95.2 19991024 (release-2)]'
perhaps checking for "GCC" in sys.version will do what we want :-))
No; as I understand it, the CygwinCCompiler class will allow you to build extensions using GCC (Cygwin or MingW32!) with the stock Python binary, as long as you have a libpython.a around. (It also requires changes to Python's config.h, but I believe Fred checked those in a few days ago.)
You are both right. I think Norman compiles it in the UNIX way using configure. In this case it builds its own config.h and this is definitively able to be used for extension building with GNU C. But if you compile it with MSVC or use the downloaded binary version, then you have this special PC config.h file. And in this file you need a special section for GNU C. (probably using #ifdef __GNUC__ somewhere.) So I will change my code, if it finds GCC in sys.version, then it doesn't need to check the config.h file. (Norman, have you already tried to build an extension? It seems you are only one out there who is also using GNU C. I'm using currently the mingw32 port, so I need someone which uses an up-to-date Cygwin. My Cygwin was an old version (2.91.67)) Kind regards Rene Liebscher
On 30 June 2000, Rene Liebscher said:
But if you compile it with MSVC or use the downloaded binary version, then you have this special PC config.h file. And in this file you need a special section for GNU C. (probably using #ifdef __GNUC__ somewhere.)
So I will change my code, if it finds GCC in sys.version, then it doesn't need to check the config.h file.
This is all to avoid the "Python's config.h doesn't seem to support your compiler." warning, right? Or is there something more to it? Have a patch yet, BTW? Greg -- Greg Ward - programmer-at-big gward@python.net http://starship.python.net/~gward/ "I want to be *different*, just like everyone ELSE!"
Greg Ward wrote:
On 30 June 2000, Rene Liebscher said:
But if you compile it with MSVC or use the downloaded binary version, then you have this special PC config.h file. And in this file you need a special section for GNU C. (probably using #ifdef __GNUC__ somewhere.)
So I will change my code, if it finds GCC in sys.version, then it doesn't need to check the config.h file.
This is all to avoid the "Python's config.h doesn't seem to support your compiler." warning, right? Or is there something more to it?
Yes, on a GCC compiled python you would get this warning.
Have a patch yet, BTW? And here is the patch. (It removes also the msvc_hack from build_ext.)
It fixes my compiler class to avoid this warning on a GCC compiled python and it removes all the python specific stuff (except the above). It inserts this in build_ext (specifying the export symbol init{module} if no export symbols given and link the library python??) This means also I had to adapt the msvc compiler class and the Extension class (It should be possible to specify None as export symbols, this would mean standard python module with one export symbol, specifying a list would mean export what is in the list.) The def-file will go through the export_symbols parameter. (as string or if given export_symbols as list) I didn't changed bcppcompiler.py because I can't test it. kind regards Rene Liebscher diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py dp/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Fri Jun 30 17:18:01 2000 +++ dp/distutils/command/build_ext.py Wed Jul 5 11:27:48 2000 @@ -420,16 +420,40 @@ objects.extend (ext.extra_objects) extra_args = ext.extra_link_args or [] - # Bunch of fixing-up we have to do for Microsoft's linker. - if self.compiler.compiler_type == 'msvc': - self.msvc_prelink_hack(sources, ext, extra_args) + # For all platforms and compiler doesn't export all symbols + # by default, we have to specify the module's init function + # if the use doesn't provide any export information. + # All other platforms and compilers should ignore such + # export information. + if ext.export_symbol_file == None and ext.export_symbols == None: + # if no exports are specified we can't build a working + # python module. + # so let us export the module's init function + export_symbols = ["init" + string.split(ext.name,'.')[-1]] + else: + # export_symbols will be a list, a string or None if + # neither export_symbols nor export_symbol_file was set. + export_symbols = ext.export_symbols or ext.export_symbol_file + + # The python library is always needed on + # Windows we need the python version without the dot, eg. '15' + # MSVC doesn't really need this, because the pragma in config.h + if sys.platform == "win32": + pythonlib = ("python%d%d" % + (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) + # don't extend ext.libraries, it may be shared with other + # extensions, it is a reference to the original list + libraries = ext.libraries + [pythonlib] + else: + libraries = ext.libraries self.compiler.link_shared_object ( objects, ext_filename, - libraries=ext.libraries, + libraries=libraries, library_dirs=ext.library_dirs, runtime_library_dirs=ext.runtime_library_dirs, extra_postargs=extra_args, + export_symbols=export_symbols, debug=self.debug, build_temp=self.build_temp) @@ -511,44 +535,6 @@ # find_swig () - - # -- Hooks 'n hacks ------------------------------------------------ - - def msvc_prelink_hack (self, sources, ext, extra_args): - - # XXX this is a kludge! Knowledge of specific compilers or - # platforms really doesn't belong here; in an ideal world, the - # CCompiler interface would provide access to everything in a - # compiler/linker system needs to build Python extensions, and - # we would just do everything nicely and cleanly through that - # interface. However, this is a not an ideal world and the - # CCompiler interface doesn't handle absolutely everything. - # Thus, kludges like this slip in occasionally. (This is no - # excuse for committing more platform- and compiler-specific - # kludges; they are to be avoided if possible!) - - def_file = ext.export_symbol_file - - if def_file is not None: - extra_args.append ('/DEF:' + def_file) - else: - modname = string.split (ext.name, '.')[-1] - extra_args.append('/export:init%s' % modname) - - # The MSVC linker generates .lib and .exp files, which cannot be - # suppressed by any linker switches. The .lib files may even be - # needed! Make sure they are generated in the temporary build - # directory. Since they have different names for debug and release - # builds, they can go into the same directory. - implib_file = os.path.join ( - self.implib_dir, - self.get_ext_libname (ext.name)) - extra_args.append ('/IMPLIB:' + implib_file) - self.mkpath (os.path.dirname (implib_file)) - - # msvc_prelink_hack () - - # -- Name generators ----------------------------------------------- # (extension names, filenames, whatever) diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/cygwinccompiler.py dp/distutils/cygwinccompiler.py --- distutils.orig/distutils/cygwinccompiler.py Fri Jun 30 17:17:55 2000 +++ dp/distutils/cygwinccompiler.py Wed Jul 5 12:29:57 2000 @@ -6,11 +6,34 @@ cygwin in no-cygwin mode). """ +# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a +# +# see also http://starship.python.net/crew/kernr/mingw32 +# +# * cygwin gcc 2.91 works (after patching some include files) +# * mingw32 gcc 2.95.2 works (it doesn't support gcc -shared) +# * cygwin gcc 2.95.2 doesn't work now +# - its dllwrap doesn't work at my machine +# - target had to be changed to something like i686-cygwin32 +# - mingw32-mode is specified by -mno-cygwin +# - using gcc -shared instead dllwrap didn't work because some problems with +# the import library, maybe it needs some additional parameters + + # created 2000/05/05, Rene Liebscher __revision__ = "$Id: cygwinccompiler.py,v 1.2 2000/06/29 22:57:55 gward Exp $" import os,sys,string +from types import * from distutils import sysconfig from distutils.unixccompiler import UnixCCompiler @@ -20,13 +43,25 @@ def check_config_h(): """Checks if the GCC compiler is mentioned in config.h. If it is not, - compiling probably doesn't work, so print a warning to stderr. + compiling probably doesn't work. """ - - # XXX the result of the check should be returned! + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK from distutils import sysconfig import string,sys + # if sys.version contains GCC then python was compiled with + # GCC, and the config.h file should be OK + try: + string.index(sys.version,"GCC") + return 2 + except ValueError: + pass # go to the next test + try: # It would probably better to read single lines to search. # But we do this only once, and it is fast enough @@ -35,19 +70,15 @@ f.close() try: # is somewhere a #ifdef __GNUC__ or something similar - string.index(s,"__GNUC__") + string.index(s,"__GNUC__") + return 1 except ValueError: - sys.stderr.write ("warning: "+ - "Python's config.h doesn't seem to support your compiler.\n") + return -1 except IOError: # if we can't read this file, we cannot say it is wrong # the compiler will complain later about this file as missing pass - - -# This is called when the module is imported, so we make this check only once -# XXX why not make it only when the compiler is needed? -check_config_h() + return 0 class CygwinCCompiler (UnixCCompiler): @@ -59,8 +90,13 @@ dry_run=0, force=0): + UnixCCompiler.__init__ (self, verbose, dry_run, force) + if check_config_h()<=0: + self.warn( + "Python's config.h doesn't seem to support your compiler.") + # Hard-code GCC because that's what this is all about. # XXX optimization, warnings etc. should be customizable. self.set_executables(compiler='gcc -O -Wall', @@ -96,53 +132,55 @@ if libraries == None: libraries = [] - # Additional libraries: the python library is always needed on - # Windows we need the python version without the dot, eg. '15' - - pythonlib = ("python%d%d" % - (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) - libraries.append(pythonlib) - libraries.extend(self.dll_libraries) - - # name of extension - - # XXX WRONG WRONG WRONG - # this is NOT the place to make guesses about Python namespaces; - # that MUST be done in build_ext.py + # Additional libraries + # libraries.extend(...) would affect the list we got as parameter + # and this can come from anywhere + libraries = libraries + self.dll_libraries + + # we want to put some files in the same directory as the + # object files are, build_temp doesn´t help much + # name of dll if not debug: - ext_name = os.path.basename(output_filename)[:-len(".pyd")] + dll_name = os.path.basename(output_filename)[:-len(".pyd")] else: - ext_name = os.path.basename(output_filename)[:-len("_d.pyd")] + dll_name = os.path.basename(output_filename)[:-len("_d.pyd")] - def_file = os.path.join(build_temp, ext_name + ".def") - #exp_file = os.path.join(build_temp, ext_name + ".exp") - #lib_file = os.path.join(build_temp, 'lib' + ext_name + ".a") + # where are the object files + temp_dir=os.path.dirname(objects[0]) + + # generate the filenames for these files + def_file = None # this will be done later, if necessary + exp_file = os.path.join(temp_dir, dll_name + ".exp") + lib_file = os.path.join(temp_dir, 'lib' + dll_name + ".a") - # Make .def file - # (It would probably better to check if we really need this, - # but for this we had to insert some unchanged parts of - # UnixCCompiler, and this is not what we want.) - f = open(def_file,"w") - f.write("EXPORTS\n") # intro - if export_symbols == None: - # export a function "init" + ext_name - f.write("init" + ext_name + "\n") + # check what we got in export_symbols + if export_symbols != None and type(export_symbols) == StringType: + def_file = export_symbol_file else: - # if there are more symbols to export write them into f - for sym in export_symbols: - f.write(sym+"\n") - f.close() - + # Make .def file + # (It would probably better to check if we really need this, + # but for this we had to insert some unchanged parts of + # UnixCCompiler, and this is not what we want.) + if type(export_symbols) == ListType : + def_file = os.path.join(temp_dir, dll_name + ".def") + f = open(def_file,"w") + f.write("EXPORTS\n") # intro + for sym in export_symbols: + f.write(sym+"\n") + f.close() + + if extra_preargs == None: extra_preargs = [] extra_preargs = extra_preargs + [ #"--verbose", - #"--output-exp",exp_file, - #"--output-lib",lib_file, - "--def",def_file + "--output-exp",exp_file, + "--output-lib",lib_file, ] + if def_file: + extra_preargs.extend(["--def",def_file]) # who wants symbols and a many times larger output file # should explicitely switch the debug mode on diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/extension.py dp/distutils/extension.py --- distutils.orig/distutils/extension.py Mon Jun 26 09:34:35 2000 +++ dp/distutils/extension.py Tue Jul 4 17:00:00 2000 @@ -111,7 +111,7 @@ self.extra_objects = extra_objects or [] self.extra_compile_args = extra_compile_args or [] self.extra_link_args = extra_link_args or [] - self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols self.export_symbol_file = export_symbol_file # class Extension diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/msvccompiler.py dp/distutils/msvccompiler.py --- distutils.orig/distutils/msvccompiler.py Mon Jul 3 12:33:44 2000 +++ dp/distutils/msvccompiler.py Tue Jul 4 18:06:50 2000 @@ -374,11 +374,25 @@ ldflags = self.ldflags_shared export_opts = [] - for sym in (export_symbols or []): - export_opts.append("/EXPORT:" + sym) + if export_symbols != None and type(export_symbols) == StringType: + export_opts.append('/DEF:' + export_symbols) + else: + for sym in (export_symbols or []): + export_opts.append("/EXPORT:" + sym) ld_args = (ldflags + lib_opts + export_opts + objects + ['/OUT:' + output_filename]) + + # The MSVC linker generates .lib and .exp files, which cannot be + # suppressed by any linker switches. The .lib files may even be + # needed! Make sure they are generated in the temporary build + # directory. Since they have different names for debug and release + # builds, they can go into the same directory. + dll_name, dll_ext = os.path.splitext(os.path.basename(output_filename)) + implib_file = os.path.join ( + os.path.dirname(objects[0]), + self.static_lib_format % (dll_name, self.static_lib_extension)) + ld_args.append ('/IMPLIB:' + implib_file) if extra_preargs: ld_args[:0] = extra_preargs
On 06 July 2000, Rene Liebscher said:
It fixes my compiler class to avoid this warning on a GCC compiled python and it removes all the python specific stuff (except the above).
OK, good.
It inserts this in build_ext (specifying the export symbol init{module} if no export symbols given and link the library python??) This means also I had to adapt the msvc compiler class and the Extension class (It should be possible to specify None as export symbols, this would mean standard python module with one export symbol, specifying a list would mean export what is in the list.)
Agreed on changed semantics of 'export_symbols'. One reservation: I don't see how putting the "implib" file in the temp dir is taken care of. I agree that it's better to do it in msvccompiler.py using the 'build_temp' parameter to the 'link_*()' methods, but I don't see that being done in your patch. Am I missing something or did you forget that?
I didn't changed bcppcompiler.py because I can't test it.
OK. Lyle, are you out there and following this? I can try to hack up bcppcompiler.py to agree with msvccompiler.py and cygwincompiler.py, but I can't test it either.
The def-file will go through the export_symbols parameter. (as string or if given export_symbols as list)
Uh no, that's wrong: that's what 'export_symbol_file' is for. Wait, maybe it's my fault for not supporting 'export_symbol_file' in the CCompiler interface. Oops. I vote we drop 'export_symbol_file' from Extension -- ie. require that the symbols to export be listed in a Python list in your setup script, rather than in an external file -- and see if anyone howls. Detailed comments on the patch...
--- distutils.orig/distutils/command/build_ext.py Fri Jun 30 17:18:01 2000 +++ dp/distutils/command/build_ext.py Wed Jul 5 11:27:48 2000 @@ -420,16 +420,40 @@ objects.extend (ext.extra_objects) extra_args = ext.extra_link_args or []
- # Bunch of fixing-up we have to do for Microsoft's linker. - if self.compiler.compiler_type == 'msvc': - self.msvc_prelink_hack(sources, ext, extra_args) + # For all platforms and compiler doesn't export all symbols + # by default, we have to specify the module's init function + # if the use doesn't provide any export information. + # All other platforms and compilers should ignore such + # export information. [...]
Aiee! Too much code in one place. Please factor this out into two new methods: I'd suggest * get_export_symbols(ext) -> [string] determine the list of symbols to export from the .so/.pyd * get_libraries(ext) -> [string] determine the list of libraries to link with when generating the .so/.pyd
- - # -- Hooks 'n hacks ------------------------------------------------ - - def msvc_prelink_hack (self, sources, ext, extra_args): - [...] - implib_file = os.path.join ( - self.implib_dir, - self.get_ext_libname (ext.name)) - extra_args.append ('/IMPLIB:' + implib_file) - self.mkpath (os.path.dirname (implib_file))
Again, I'm not convinced we can get rid of those yet. Convince me. ;-)
--- distutils.orig/distutils/cygwinccompiler.py Fri Jun 30 17:17:55 2000 +++ dp/distutils/cygwinccompiler.py Wed Jul 5 12:29:57 2000 @@ -6,11 +6,34 @@ cygwin in no-cygwin mode). """
+# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a
Ouch! So the Cygwin support is definitely not ready for prime time with 1.5.2. My understanding is that #1 has been fixed in the current Python 2.0 CVS code. What about #2? Should we try to convince Guido to include libpython20.a the 2.0 distribution to save Cygwin/Mingw32 users the bother of generating it? Or maybe "build_ext" can grow a little bit of compiler-sensitivity for Cygwin/Mingw32 and run the above command to generate libpython20.a the first time the user tries to build an extension with Cygwin/Mingw32. Come to think of it, this solves the problem nicely for 1.5.2. And hey, we could even do the same thing for config.h, but that's rather more dangerous -- modifying an existing file rather than generating a new one. Worth considering, though.
+# * cygwin gcc 2.91 works (after patching some include files) +# * mingw32 gcc 2.95.2 works (it doesn't support gcc -shared) +# * cygwin gcc 2.95.2 doesn't work now +# - its dllwrap doesn't work at my machine +# - target had to be changed to something like i686-cygwin32 +# - mingw32-mode is specified by -mno-cygwin +# - using gcc -shared instead dllwrap didn't work because some problems with +# the import library, maybe it needs some additional parameters
AAUUGGHH!!! (What more can I say?)
def check_config_h(): """Checks if the GCC compiler is mentioned in config.h. If it is not, - compiling probably doesn't work, so print a warning to stderr. + compiling probably doesn't work. """ - - # XXX the result of the check should be returned! + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK
Ick. Magic numbers bad. Strings good. Intelligent exception objects (often) better. Think about it.
+ # if sys.version contains GCC then python was compiled with + # GCC, and the config.h file should be OK + try: + string.index(sys.version,"GCC") + return 2 + except ValueError: + pass # go to the next test +
I don't like relying on exception-handling like this: I think I'm superstitious about the cost of invoking an unnecessary exception. Please recode using string.find().
+ UnixCCompiler.__init__ (self, verbose, dry_run, force)
+ if check_config_h()<=0: + self.warn( + "Python's config.h doesn't seem to support your compiler.")
Should this warning add something like, "expect compilation to die horribly"?
+ # name of dll if not debug: - ext_name = os.path.basename(output_filename)[:-len(".pyd")] + dll_name = os.path.basename(output_filename)[:-len(".pyd")] else: - ext_name = os.path.basename(output_filename)[:-len("_d.pyd")] + dll_name = os.path.basename(output_filename)[:-len("_d.pyd")]
More Python-specific stuff. Note that all this is already done for you in build_ext.py: see 'get_ext_filename()'. As long as it's being called appropriately, the compiler class doesn't need to figure out what the name of the file it's going to create.
+ # where are the object files + temp_dir=os.path.dirname(objects[0])
What about the 'build_temp' parameter?
+ if export_symbols != None and type(export_symbols) == StringType: + def_file = export_symbol_file
Minor quibble: I use "is" for comparing to None and type objects. I know it makes no difference, but that's the Distutils Coding Standard. So there. ;-) This concern will go away if we stop supporting 'export_symbol_file', but 'export_symbols' should always be a list. I hate the "if it's a string, it's a filename, if it's a list, it's a list of symbols" distinction. Makes the code tricky to follow.
+ # Make .def file + # (It would probably better to check if we really need this, + # but for this we had to insert some unchanged parts of + # UnixCCompiler, and this is not what we want.) + if type(export_symbols) == ListType : + def_file = os.path.join(temp_dir, dll_name + ".def") + f = open(def_file,"w") + f.write("EXPORTS\n") # intro + for sym in export_symbols: + f.write(sym+"\n") + f.close()
Is it necessary to make a .def file if there are multiple export symbols? Or can they go on the command line? (I think I'd rather put them on the command-line: more explicit, and one less temp file to worry about.)
- self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols
I understand the reason for this change, but we need to make sure there isn't code out there that relies on export_symbols always being a list (ie. never None).
--- distutils.orig/distutils/msvccompiler.py Mon Jul 3 12:33:44 2000 +++ dp/distutils/msvccompiler.py Tue Jul 4 18:06:50 2000 @@ -374,11 +374,25 @@ [...] + if export_symbols != None and type(export_symbols) == StringType: + export_opts.append('/DEF:' + export_symbols)
"is" again, and again I express distaste for this sort of type-sensitivity.
+ # The MSVC linker generates .lib and .exp files, which cannot be + # suppressed by any linker switches. The .lib files may even be + # needed! Make sure they are generated in the temporary build + # directory. Since they have different names for debug and release + # builds, they can go into the same directory. + dll_name, dll_ext = os.path.splitext(os.path.basename(output_filename)) + implib_file = os.path.join ( + os.path.dirname(objects[0]), + self.static_lib_format % (dll_name, self.static_lib_extension)) + ld_args.append ('/IMPLIB:' + implib_file)
Ahh, is this the code that's supposed to replace the last stanza of 'msvc_prelink_hack()'? If so, why isn't it putting the "implib" file in 'build_temp'? ...oh of course, it goes there anyways, because that's where object files go. This just respects the convention of preserving the source tree structure under build_temp. I take it all back; I think this is right. Can you regenerate your patch to take the above comments into account and resubmit it? Thanks a bunch! Greg -- Greg Ward - just another Python hacker gward@python.net http://starship.python.net/~gward/ All of life is a blur of Republicans and meat!
Greg Ward wrote:
On 06 July 2000, Rene Liebscher said:
I didn't changed bcppcompiler.py because I can't test it.
OK. Lyle, are you out there and following this? I can try to hack up bcppcompiler.py to agree with msvccompiler.py and cygwincompiler.py, but I can't test it either. But I can test it now. I found the problem Lyle had in it. (He forgot to close the file in which he wrote the export symbols, and the compiler didn't want use an open file.) I also added some comments and a extension concerning libraries. It now looks first if there is a library called bcpp_{libname}.lib before it tries {libname}.lib. So you can have both,
the default msvc libraries and special libraries for Borland C++. (They are not compatible.) This is useful if you have installed your Python in a network and user use different compilers. (Or for someone like me, who has now six different compilers on one machine to test distutils compiler classes.)
I vote we drop 'export_symbol_file' from Extension -- ie. require that the symbols to export be listed in a Python list in your setup script, rather than in an external file -- and see if anyone howls. It is your decision, export_symbol_file is now no longer used anywhere, but it still exists as parameter to Extension.
Detailed comments on the patch...
--- distutils.orig/distutils/cygwinccompiler.py Fri Jun 30 17:17:55 2000 +++ dp/distutils/cygwinccompiler.py Wed Jul 5 12:29:57 2000 @@ -6,11 +6,34 @@ cygwin in no-cygwin mode). """
+# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a
Ouch! So the Cygwin support is definitely not ready for prime time with 1.5.2. My understanding is that #1 has been fixed in the current Python 2.0 CVS code. Not yet, but there is a patch in SourceForges open patches list.
What about #2? Should we try to convince Guido to include libpython20.a the 2.0 distribution to save Cygwin/Mingw32 users the bother of generating it?
Or maybe "build_ext" can grow a little bit of compiler-sensitivity for Cygwin/Mingw32 and run the above command to generate libpython20.a the first time the user tries to build an extension with Cygwin/Mingw32. Come to think of it, this solves the problem nicely for 1.5.2. And hey, we could even do the same thing for config.h, but that's rather more dangerous -- modifying an existing file rather than generating a new one. Worth considering, though. I think he should add a def-file for python20.dll in the distribution. Every serious compiler can build from this an import library. (Many can also use the dll to build it, but most fail because Python exports not only functions, it also exports data.) (Borland can also translate MSVC import libraries into its own format.)
We could also add a tools directory or something similar. In this directory we could put some python programs which generate an import library for a given def-file, library or dll. They could called ????_def2lib.py, ????_lib2lib.py and ????_dll2lib.py, where ???? stands for the compiler (bcpp,...). This would help users to use other dlls or libraries they want to use for extension. This would also the place to put a replacement for the python 1.5.2 config.h file. I think the python 2.0 config.h should work fine, after changing references from 2.0 to 1.5 (#pragma comment(lib,"python20.lib")
+# * cygwin gcc 2.91 works (after patching some include files) +# * mingw32 gcc 2.95.2 works (it doesn't support gcc -shared) +# * cygwin gcc 2.95.2 doesn't work now +# - its dllwrap doesn't work at my machine +# - target had to be changed to something like i686-cygwin32 +# - mingw32-mode is specified by -mno-cygwin +# - using gcc -shared instead dllwrap didn't work because some problems with +# the import library, maybe it needs some additional parameters
AAUUGGHH!!! (What more can I say?)
- # XXX the result of the check should be returned! + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK
Ick. Magic numbers bad. Strings good. Intelligent exception objects (often) better. Think about it. Not magic numbers, more a kind of level, higher number mean a greater
gcc 2.91 is no problem for us. People should use a newer compiler. But the differences betweeen these two gcc 2.9.5 are a problem, mingw32 doesn't support gcc -shared yet, and cygwin's dllwrap seems to have a bug (It fails to reallocate 1.7 GByte of Memory.) Cygwin's gcc -shared wants other symbols than dlltool creates in the import library. Is there someone out there who has already successful created a extension module with the new cygwin version? If yes, please send me the commandlines you used for this. probability that compiling will successful.
Is it necessary to make a .def file if there are multiple export symbols? Or can they go on the command line? (I think I'd rather put them on the command-line: more explicit, and one less temp file to worry about.)
Normally gcc exports all symbols if you don't specify any exports related options. But you can't rely on it, I also found some extensions build without exports. This why we have to specify them. There is an exclude-symbols option, but I never found a export-symbols or include-symbols or something similar. It seems we have to build this def-file.
- self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols
I understand the reason for this change, but we need to make sure there isn't code out there that relies on export_symbols always being a list (ie. never None). It was never used by build_ext and in the compiler interface it has a default value of None. So it is unlikely someone will even take notice of it.
And now something new. Is anyone out there using the LCC-Win32 compiler? (http://www.cs.virginia.edu/~lcc-win32/) It is a small C compiler, but it comes with a IDE (gcc and free BCC55 don't) and it is able to build python extensions. I have a compiler class for it, which is already able to build extensions. So if anyone is interested or maybe want it to complete, I will post it to the mailing list. kind regards Rene Liebscher diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/bcppcompiler.py distutils.patched/distutils/bcppcompiler.py --- distutils.orig/distutils/bcppcompiler.py Wed Jun 28 03:20:35 2000 +++ distutils.patched/distutils/bcppcompiler.py Wed Jul 12 14:33:34 2000 @@ -11,17 +11,6 @@ # someone should sit down and factor out the common code as # WindowsCCompiler! --GPW -# XXX Lyle reports that this doesn't quite work yet: -# """...but this is what I've got so far. The compile step works fine but -# when it runs the link step I get an "out of memory" failure. Since -# spawn() echoes the command it's trying to spawn, I can type the link line -# verbatim at the DOS prompt and it links the Windows DLL correctly -- so -# the syntax is correct. There's just some weird interaction going on when -# it tries to "spawn" the link process from within the setup.py script. I'm -# not really sure how to debug this one right off-hand; obviously there's -# nothing wrong with the "spawn()" function since it's working properly for -# the compile stage.""" - __revision__ = "$Id: bcppcompiler.py,v 1.1 2000/06/28 01:20:35 gward Exp $" @@ -123,6 +112,9 @@ elif ext in self._cpp_extensions: input_opt = "-P" + src = os.path.normpath(src) + obj = os.path.normpath(obj) + output_opt = "-o" + obj self.mkpath (os.path.dirname (obj)) @@ -230,24 +222,55 @@ else: ldflags = self.ldflags_shared + # Borland C++ has problems with '/' in paths + objects = map(os.path.normpath,objects) + startup_obj = 'c0d32' - libraries.append ('mypylib') + # either exchange python15.lib in the python libs directory against + # a Borland-like one, or create one with name bcpp_python15.lib + # there and remove the pragmas from config.h + #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt') # Create a temporary exports file for use by the linker head, tail = os.path.split (output_filename) modname, ext = os.path.splitext (tail) - def_file = os.path.join (build_temp, '%s.def' % modname) + temp_dir = os.path.dirname(objects[0]) # preserve tree structure + def_file = os.path.join (temp_dir, '%s.def' % modname) f = open (def_file, 'w') f.write ('EXPORTS\n') for sym in (export_symbols or []): f.write (' %s=_%s\n' % (sym, sym)) - - ld_args = ldflags + [startup_obj] + objects + \ - [',%s,,' % output_filename] + \ - libraries + [',' + def_file] + f.close() + + # start building command line + # flags and options + ld_args = list(ldflags) # don't use a reference, use a separate copy + for l in library_dirs: + ld_args.append("/L%s" % os.path.normpath(l)) + # objects list + ld_args.extend([startup_obj]) + ld_args.extend(objects) + # name of dll file + ld_args.extend([',',output_filename]) + # no map file + ld_args.extend([',']) + # start libraries + ld_args.extend([',']) + for lib in libraries: + # see if we find it and if there is a bcpp specific lib (bcpp_xxx.lib) + l = self.find_library_file(library_dirs,lib,debug) + if l == None: + ld_args.append(lib) + # probably an bcpp internal library + # self.warn('library %s not found.' % lib) + else: + # full name which prefers bcpp_xxx.lib over xxx.lib + ld_args.append(l) + # def file for export symbols + ld_args.extend([',',def_file]) if extra_preargs: ld_args[:0] = extra_preargs @@ -325,15 +348,28 @@ def runtime_library_dir_option (self, dir): raise DistutilsPlatformError, \ - "don't know how to set runtime library search path for MSVC++" + "don't know how to set runtime library search path for Borland C++" def library_option (self, lib): return self.library_filename (lib) - def find_library_file (self, dirs, lib): - + def find_library_file (self, dirs, lib, debug=0): + # find library file + # bcpp_xxx.lib is better than xxx.lib + # and xxx_d.lib is better than xxx.lib if debug is set for dir in dirs: + if debug: + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib + "_d")) + if os.path.exists (libfile): + return libfile + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib)) + if os.path.exists (libfile): + return libfile + if debug: + libfile = os.path.join (dir, self.library_filename (lib + '_d')) + if os.path.exists (libfile): + return libfile libfile = os.path.join (dir, self.library_filename (lib)) if os.path.exists (libfile): return libfile diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py distutils.patched/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Fri Jul 7 22:41:21 2000 +++ distutils.patched/distutils/command/build_ext.py Wed Jul 12 13:57:48 2000 @@ -283,6 +283,9 @@ # Medium-easy stuff: same syntax/semantics, different names. ext.runtime_library_dirs = build_info.get('rpath') ext.export_symbol_file = build_info.get('def_file') + if ext.export_symbol_file is not None: + self.warn("export_symbol_file is no longer supported, " + "comments and questions to <distutils-sig@python.org>") # Non-trivial stuff: 'macros' split into 'define_macros' # and 'undef_macros'. @@ -420,16 +423,14 @@ objects.extend (ext.extra_objects) extra_args = ext.extra_link_args or [] - # Bunch of fixing-up we have to do for Microsoft's linker. - if self.compiler.compiler_type == 'msvc': - self.msvc_prelink_hack(sources, ext, extra_args) self.compiler.link_shared_object ( objects, ext_filename, - libraries=ext.libraries, + libraries=self.get_libraries(ext), library_dirs=ext.library_dirs, runtime_library_dirs=ext.runtime_library_dirs, extra_postargs=extra_args, + export_symbols=self.get_export_symbols(ext), debug=self.debug, build_temp=self.build_temp) @@ -511,44 +512,6 @@ # find_swig () - - # -- Hooks 'n hacks ------------------------------------------------ - - def msvc_prelink_hack (self, sources, ext, extra_args): - - # XXX this is a kludge! Knowledge of specific compilers or - # platforms really doesn't belong here; in an ideal world, the - # CCompiler interface would provide access to everything in a - # compiler/linker system needs to build Python extensions, and - # we would just do everything nicely and cleanly through that - # interface. However, this is a not an ideal world and the - # CCompiler interface doesn't handle absolutely everything. - # Thus, kludges like this slip in occasionally. (This is no - # excuse for committing more platform- and compiler-specific - # kludges; they are to be avoided if possible!) - - def_file = ext.export_symbol_file - - if def_file is not None: - extra_args.append ('/DEF:' + def_file) - else: - modname = string.split (ext.name, '.')[-1] - extra_args.append('/export:init%s' % modname) - - # The MSVC linker generates .lib and .exp files, which cannot be - # suppressed by any linker switches. The .lib files may even be - # needed! Make sure they are generated in the temporary build - # directory. Since they have different names for debug and release - # builds, they can go into the same directory. - implib_file = os.path.join ( - self.implib_dir, - self.get_ext_libname (ext.name)) - extra_args.append ('/IMPLIB:' + implib_file) - self.mkpath (os.path.dirname (implib_file)) - - # msvc_prelink_hack () - - # -- Name generators ----------------------------------------------- # (extension names, filenames, whatever) @@ -578,5 +541,33 @@ if os.name == 'nt' and self.debug: return apply (os.path.join, ext_path) + '_d.lib' return apply (os.path.join, ext_path) + '.lib' + + + def get_export_symbols (self, ext): + # For all platforms and compiler doesn't export all symbols + # by default, we have to specify the module's init function + # if the user doesn't provide any export information. + # All other platforms and compilers should ignore such + # export information. + if ext.export_symbols is None: + # if no exports are specified we can't build a working + # python module. + # so let us export the module's init function + return ["init" + string.split(ext.name,'.')[-1]] + else: + return ext.export_symbols + + def get_libraries (self, ext): + # The python library is always needed on Windows. + # We need the python version without the dot, eg. '15' + # MSVC doesn't really need this, because the pragma in config.h + if sys.platform == "win32": + pythonlib = ("python%d%d" % + (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) + # don't extend ext.libraries, it may be shared with other + # extensions, it is a reference to the original list + return ext.libraries + [pythonlib] + else: + return ext.libraries # class build_ext diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/cygwinccompiler.py distutils.patched/distutils/cygwinccompiler.py --- distutils.orig/distutils/cygwinccompiler.py Fri Jun 30 00:57:55 2000 +++ distutils.patched/distutils/cygwinccompiler.py Wed Jul 12 14:17:57 2000 @@ -10,6 +10,29 @@ __revision__ = "$Id: cygwinccompiler.py,v 1.2 2000/06/29 22:57:55 gward Exp $" +# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a +# +# see also http://starship.python.net/crew/kernr/mingw32 +# +# * cygwin gcc 2.91 works (after patching some include files) +# * mingw32 gcc 2.95.2 works (it doesn't support gcc -shared) +# * cygwin gcc 2.95.2 doesn't work now +# - its dllwrap doesn't work at my machine +# - using gcc -shared instead dllwrap didn't work because some problems with +# the import library, maybe it needs some additional parameters. +# It tries to import _imp__XXX but dlltool creates an import library +# with __imp__XXX symbols. + + + import os,sys,string from distutils import sysconfig from distutils.unixccompiler import UnixCCompiler @@ -20,34 +43,41 @@ def check_config_h(): """Checks if the GCC compiler is mentioned in config.h. If it is not, - compiling probably doesn't work, so print a warning to stderr. + compiling probably doesn't work. """ - - # XXX the result of the check should be returned! + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK from distutils import sysconfig import string,sys + # if sys.version contains GCC then python was compiled with + # GCC, and the config.h file should be OK + if -1 == string.find(sys.version,"GCC"): + pass # go to the next test + else: + return 2 + try: # It would probably better to read single lines to search. # But we do this only once, and it is fast enough f=open(sysconfig.get_config_h_filename()) s=f.read() f.close() - try: - # is somewhere a #ifdef __GNUC__ or something similar - string.index(s,"__GNUC__") - except ValueError: - sys.stderr.write ("warning: "+ - "Python's config.h doesn't seem to support your compiler.\n") + + # is somewhere a #ifdef __GNUC__ or something similar + if -1 == string.find(s,"__GNUC__"): + return -1 + else: + return 1 except IOError: # if we can't read this file, we cannot say it is wrong # the compiler will complain later about this file as missing pass - - -# This is called when the module is imported, so we make this check only once -# XXX why not make it only when the compiler is needed? -check_config_h() + return 0 class CygwinCCompiler (UnixCCompiler): @@ -59,14 +89,20 @@ dry_run=0, force=0): + UnixCCompiler.__init__ (self, verbose, dry_run, force) + if check_config_h()<=0: + self.warn( + "Python's config.h doesn't seem to support your compiler. " + "Compiling may fail because of undefined preprocessor macros.") + # Hard-code GCC because that's what this is all about. # XXX optimization, warnings etc. should be customizable. self.set_executables(compiler='gcc -O -Wall', - compiler_so='gcc -O -Wall', + compiler_so='gcc -O -Wall -mcygwin', linker_exe='gcc', - linker_so='dllwrap --target=i386-cygwin32') + linker_so='dllwrap -mcygwin') # cygwin and mingw32 need different sets of libraries self.dll_libraries=[ @@ -96,53 +132,49 @@ if libraries == None: libraries = [] - # Additional libraries: the python library is always needed on - # Windows we need the python version without the dot, eg. '15' - - pythonlib = ("python%d%d" % - (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) - libraries.append(pythonlib) - libraries.extend(self.dll_libraries) - - # name of extension + # Additional libraries + # libraries.extend(...) would affect the list we got as parameter + # and this can come from anywhere + libraries = libraries + self.dll_libraries + + # we want to put some files in the same directory as the + # object files are, build_temp doesn´t help much - # XXX WRONG WRONG WRONG - # this is NOT the place to make guesses about Python namespaces; - # that MUST be done in build_ext.py + # name of dll to give the helper files (def, lib, exp) the same name + dll_name,dll_extension = os.path.splitext(os.path.basename(output_filename)) - if not debug: - ext_name = os.path.basename(output_filename)[:-len(".pyd")] - else: - ext_name = os.path.basename(output_filename)[:-len("_d.pyd")] + # where are the object files + temp_dir = os.path.dirname(objects[0]) - def_file = os.path.join(build_temp, ext_name + ".def") - #exp_file = os.path.join(build_temp, ext_name + ".exp") - #lib_file = os.path.join(build_temp, 'lib' + ext_name + ".a") + # generate the filenames for these files + def_file = None # this will be done later, if necessary + exp_file = os.path.join(temp_dir, dll_name + ".exp") + lib_file = os.path.join(temp_dir, 'lib' + dll_name + ".a") - # Make .def file - # (It would probably better to check if we really need this, - # but for this we had to insert some unchanged parts of - # UnixCCompiler, and this is not what we want.) - f = open(def_file,"w") - f.write("EXPORTS\n") # intro - if export_symbols == None: - # export a function "init" + ext_name - f.write("init" + ext_name + "\n") - else: - # if there are more symbols to export write them into f + # check what we got in export_symbols + if export_symbols is not None: + # Make .def file + # (It would probably better to check if we really need this, + # but for this we had to insert some unchanged parts of + # UnixCCompiler, and this is not what we want.) + def_file = os.path.join(temp_dir, dll_name + ".def") + f = open(def_file,"w") + f.write("EXPORTS\n") # intro for sym in export_symbols: f.write(sym+"\n") - f.close() - - if extra_preargs == None: + f.close() + + + if extra_preargs is None: extra_preargs = [] extra_preargs = extra_preargs + [ #"--verbose", #"--output-exp",exp_file, #"--output-lib",lib_file, - "--def",def_file ] + if def_file: + extra_preargs.extend(["--def",def_file]) # who wants symbols and a many times larger output file # should explicitely switch the debug mode on @@ -150,7 +182,7 @@ # (On my machine unstripped_file = stripped_file + 254KB # 10KB < stripped_file < ??100KB ) if not debug: - extra_preargs = extra_preargs + ["-s"] + extra_preargs.append("-s") UnixCCompiler.link_shared_object(self, objects, @@ -185,15 +217,13 @@ self.set_executables(compiler='gcc -mno-cygwin -O -Wall', compiler_so='gcc -mno-cygwin -O -Wall', linker_exe='gcc -mno-cygwin', - linker_so='dllwrap' - + ' --target=i386-mingw32' - + ' --entry _DllMain@12') - # mingw32 doesn't really need 'target' and cygwin too (it seems, - # it is enough to specify a different entry point) + linker_so='dllwrap -mno-cygwin -mthreads') + #+ ' --entry _DllMain@12') + # mingw32-mode on cygwin needs to specify a different entry point # no additional libraries need # (only msvcrt, which is already added by CygwinCCompiler) - # __init__ () - + # __init__ () + # class Mingw32CCompiler diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/extension.py distutils.patched/distutils/extension.py --- distutils.orig/distutils/extension.py Sat Jun 24 02:18:24 2000 +++ distutils.patched/distutils/extension.py Wed Jul 12 14:01:03 2000 @@ -111,7 +111,7 @@ self.extra_objects = extra_objects or [] self.extra_compile_args = extra_compile_args or [] self.extra_link_args = extra_link_args or [] - self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols self.export_symbol_file = export_symbol_file # class Extension diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/msvccompiler.py distutils.patched/distutils/msvccompiler.py --- distutils.orig/distutils/msvccompiler.py Fri Jun 30 21:37:59 2000 +++ distutils.patched/distutils/msvccompiler.py Wed Jul 12 14:01:03 2000 @@ -380,6 +380,17 @@ ld_args = (ldflags + lib_opts + export_opts + objects + ['/OUT:' + output_filename]) + # The MSVC linker generates .lib and .exp files, which cannot be + # suppressed by any linker switches. The .lib files may even be + # needed! Make sure they are generated in the temporary build + # directory. Since they have different names for debug and release + # builds, they can go into the same directory. + dll_name, dll_ext = os.path.splitext(os.path.basename(output_filename)) + implib_file = os.path.join ( + os.path.dirname(objects[0]), + self.static_lib_format % (dll_name, self.static_lib_extension)) + ld_args.append ('/IMPLIB:' + implib_file) + if extra_preargs: ld_args[:0] = extra_preargs if extra_postargs:
Rene Liebscher wrote:
Greg Ward wrote:
On 06 July 2000, Rene Liebscher said:
+# * cygwin gcc 2.91 works (after patching some include files) +# * mingw32 gcc 2.95.2 works (it doesn't support gcc -shared) +# * cygwin gcc 2.95.2 doesn't work now +# - its dllwrap doesn't work at my machine +# - target had to be changed to something like i686-cygwin32 +# - mingw32-mode is specified by -mno-cygwin +# - using gcc -shared instead dllwrap didn't work because some problems with +# the import library, maybe it needs some additional parameters
AAUUGGHH!!! (What more can I say?)
gcc 2.91 is no problem for us. People should use a newer compiler. But the differences betweeen these two gcc 2.9.5 are a problem, mingw32 doesn't support gcc -shared yet, and cygwin's dllwrap seems to have a bug (It fails to reallocate 1.7 GByte of Memory.) Cygwin's gcc -shared wants other symbols than dlltool creates in the import library. Is there someone out there who has already successful created a extension module with the new cygwin version? If yes, please send me the commandlines you used for this.
In the latest cygwin binutils 2.10.90 is a bug in dllwrap. This new version of my patch tries now to find out the version numbers of gcc, ld and dllwrap, so we can inform the user if he is using a broken version of one of these programs. (I also found many tabs in distutils which were introduced by by my help-option patch. This patch replaces these tabs with spaces.) kind regards Rene Liebscher diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/bcppcompiler.py distutils.patched/distutils/bcppcompiler.py --- distutils.orig/distutils/bcppcompiler.py Wed Jun 28 03:20:35 2000 +++ distutils.patched/distutils/bcppcompiler.py Tue Jul 18 11:20:23 2000 @@ -11,17 +11,6 @@ # someone should sit down and factor out the common code as # WindowsCCompiler! --GPW -# XXX Lyle reports that this doesn't quite work yet: -# """...but this is what I've got so far. The compile step works fine but -# when it runs the link step I get an "out of memory" failure. Since -# spawn() echoes the command it's trying to spawn, I can type the link line -# verbatim at the DOS prompt and it links the Windows DLL correctly -- so -# the syntax is correct. There's just some weird interaction going on when -# it tries to "spawn" the link process from within the setup.py script. I'm -# not really sure how to debug this one right off-hand; obviously there's -# nothing wrong with the "spawn()" function since it's working properly for -# the compile stage.""" - __revision__ = "$Id: bcppcompiler.py,v 1.1 2000/06/28 01:20:35 gward Exp $" @@ -123,6 +112,9 @@ elif ext in self._cpp_extensions: input_opt = "-P" + src = os.path.normpath(src) + obj = os.path.normpath(obj) + output_opt = "-o" + obj self.mkpath (os.path.dirname (obj)) @@ -230,24 +222,56 @@ else: ldflags = self.ldflags_shared + # Borland C++ has problems with '/' in paths + objects = map(os.path.normpath,objects) + startup_obj = 'c0d32' - libraries.append ('mypylib') + # either exchange python15.lib in the python libs directory against + # a Borland-like one, or create one with name bcpp_python15.lib + # there and remove the pragmas from config.h + #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt') # Create a temporary exports file for use by the linker head, tail = os.path.split (output_filename) modname, ext = os.path.splitext (tail) - def_file = os.path.join (build_temp, '%s.def' % modname) + temp_dir = os.path.dirname(objects[0]) # preserve tree structure + def_file = os.path.join (temp_dir, '%s.def' % modname) f = open (def_file, 'w') f.write ('EXPORTS\n') for sym in (export_symbols or []): f.write (' %s=_%s\n' % (sym, sym)) + f.close() - ld_args = ldflags + [startup_obj] + objects + \ - [',%s,,' % output_filename] + \ - libraries + [',' + def_file] + # start building command line + # flags and options + ld_args = list(ldflags) # don't use a reference, use a separate copy + for l in library_dirs: + ld_args.append("/L%s" % os.path.normpath(l)) + # objects list + ld_args.extend([startup_obj]) + ld_args.extend(objects) + # name of dll file + ld_args.extend([',',output_filename]) + # no map file + ld_args.extend([',']) + # start libraries + ld_args.extend([',']) + for lib in libraries: + # see if we find it and if there is a bcpp specific lib + # (bcpp_xxx.lib) + l = self.find_library_file(library_dirs,lib,debug) + if l == None: + ld_args.append(lib) + # probably an bcpp internal library + # self.warn('library %s not found.' % lib) + else: + # full name which prefers bcpp_xxx.lib over xxx.lib + ld_args.append(l) + # def file for export symbols + ld_args.extend([',',def_file]) if extra_preargs: ld_args[:0] = extra_preargs @@ -325,15 +349,28 @@ def runtime_library_dir_option (self, dir): raise DistutilsPlatformError, \ - "don't know how to set runtime library search path for MSVC++" + "don't know how to set runtime library search path for Borland C++" def library_option (self, lib): return self.library_filename (lib) - def find_library_file (self, dirs, lib): - + def find_library_file (self, dirs, lib, debug=0): + # find library file + # bcpp_xxx.lib is better than xxx.lib + # and xxx_d.lib is better than xxx.lib if debug is set for dir in dirs: + if debug: + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib + "_d")) + if os.path.exists (libfile): + return libfile + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib)) + if os.path.exists (libfile): + return libfile + if debug: + libfile = os.path.join (dir, self.library_filename (lib + '_d')) + if os.path.exists (libfile): + return libfile libfile = os.path.join (dir, self.library_filename (lib)) if os.path.exists (libfile): return libfile diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/bdist.py distutils.patched/distutils/command/bdist.py --- distutils.orig/distutils/command/bdist.py Wed Jul 5 05:07:18 2000 +++ distutils.patched/distutils/command/bdist.py Tue Jul 18 11:20:23 2000 @@ -42,7 +42,7 @@ help_options = [ ('help-formats', None, "lists available distribution formats", show_formats), - ] + ] # The following commands do not take a format option from bdist no_format_option = ('bdist_rpm',) diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build.py distutils.patched/distutils/command/build.py --- distutils.orig/distutils/command/build.py Sat Jun 24 03:23:37 2000 +++ distutils.patched/distutils/command/build.py Tue Jul 18 11:20:23 2000 @@ -45,7 +45,7 @@ help_options = [ ('help-compiler', None, "list available compilers", show_compilers), - ] + ] def initialize_options (self): self.build_base = 'build' diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_clib.py distutils.patched/distutils/command/build_clib.py --- distutils.orig/distutils/command/build_clib.py Sun Jun 25 04:10:58 2000 +++ distutils.patched/distutils/command/build_clib.py Tue Jul 18 11:20:23 2000 @@ -51,7 +51,7 @@ help_options = [ ('help-compiler', None, "list available compilers", show_compilers), - ] + ] def initialize_options (self): self.build_clib = None diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py distutils.patched/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Sun Jul 16 16:10:00 2000 +++ distutils.patched/distutils/command/build_ext.py Tue Jul 18 11:20:23 2000 @@ -84,7 +84,7 @@ help_options = [ ('help-compiler', None, "list available compilers", show_compilers), - ] + ] def initialize_options (self): self.extensions = None @@ -283,6 +283,9 @@ # Medium-easy stuff: same syntax/semantics, different names. ext.runtime_library_dirs = build_info.get('rpath') ext.export_symbol_file = build_info.get('def_file') + if ext.export_symbol_file is not None: + self.warn("export_symbol_file is no longer supported, " + "comments and questions to <distutils-sig@python.org>") # Non-trivial stuff: 'macros' split into 'define_macros' # and 'undef_macros'. @@ -420,16 +423,14 @@ objects.extend (ext.extra_objects) extra_args = ext.extra_link_args or [] - # Bunch of fixing-up we have to do for Microsoft's linker. - if self.compiler.compiler_type == 'msvc': - self.msvc_prelink_hack(sources, ext, extra_args) self.compiler.link_shared_object ( objects, ext_filename, - libraries=ext.libraries, + libraries=self.get_libraries(ext), library_dirs=ext.library_dirs, runtime_library_dirs=ext.runtime_library_dirs, extra_postargs=extra_args, + export_symbols=self.get_export_symbols(ext), debug=self.debug, build_temp=self.build_temp) @@ -511,44 +512,6 @@ # find_swig () - - # -- Hooks 'n hacks ------------------------------------------------ - - def msvc_prelink_hack (self, sources, ext, extra_args): - - # XXX this is a kludge! Knowledge of specific compilers or - # platforms really doesn't belong here; in an ideal world, the - # CCompiler interface would provide access to everything in a - # compiler/linker system needs to build Python extensions, and - # we would just do everything nicely and cleanly through that - # interface. However, this is a not an ideal world and the - # CCompiler interface doesn't handle absolutely everything. - # Thus, kludges like this slip in occasionally. (This is no - # excuse for committing more platform- and compiler-specific - # kludges; they are to be avoided if possible!) - - def_file = ext.export_symbol_file - - if def_file is not None: - extra_args.append ('/DEF:' + def_file) - else: - modname = string.split (ext.name, '.')[-1] - extra_args.append('/export:init%s' % modname) - - # The MSVC linker generates .lib and .exp files, which cannot be - # suppressed by any linker switches. The .lib files may even be - # needed! Make sure they are generated in the temporary build - # directory. Since they have different names for debug and release - # builds, they can go into the same directory. - implib_file = os.path.join ( - self.implib_dir, - self.get_ext_libname (ext.name)) - extra_args.append ('/IMPLIB:' + implib_file) - self.mkpath (os.path.dirname (implib_file)) - - # msvc_prelink_hack () - - # -- Name generators ----------------------------------------------- # (extension names, filenames, whatever) @@ -578,5 +541,33 @@ if os.name == 'nt' and self.debug: return apply (os.path.join, ext_path) + '_d.lib' return apply (os.path.join, ext_path) + '.lib' + + + def get_export_symbols (self, ext): + # For all platforms and compiler doesn't export all symbols + # by default, we have to specify the module's init function + # if the user doesn't provide any export information. + # All other platforms and compilers should ignore such + # export information. + if ext.export_symbols is None: + # if no exports are specified we can't build a working + # python module. + # so let us export the module's init function + return ["init" + string.split(ext.name,'.')[-1]] + else: + return ext.export_symbols + + def get_libraries (self, ext): + # The python library is always needed on Windows. + # We need the python version without the dot, eg. '15' + # MSVC doesn't really need this, because the pragma in config.h + if sys.platform == "win32": + pythonlib = ("python%d%d" % + (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) + # don't extend ext.libraries, it may be shared with other + # extensions, it is a reference to the original list + return ext.libraries + [pythonlib] + else: + return ext.libraries # class build_ext diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/install_data.py distutils.patched/distutils/command/install_data.py --- distutils.orig/distutils/command/install_data.py Sat Jun 24 19:36:24 2000 +++ distutils.patched/distutils/command/install_data.py Tue Jul 18 11:20:23 2000 @@ -32,9 +32,9 @@ def finalize_options (self): self.set_undefined_options('install', - ('install_data', 'install_dir'), - ('root', 'root'), - ) + ('install_data', 'install_dir'), + ('root', 'root'), + ) def run (self): self.mkpath(self.install_dir) diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/install_lib.py distutils.patched/distutils/command/install_lib.py --- distutils.orig/distutils/command/install_lib.py Sat May 27 19:27:23 2000 +++ distutils.patched/distutils/command/install_lib.py Tue Jul 18 11:20:23 2000 @@ -74,7 +74,7 @@ out_fn = f + (__debug__ and "c" or "o") compile_msg = "byte-compiling %s to %s" % \ (f, os.path.basename (out_fn)) - skip_msg = "byte-compilation of %s skipped" % f + skip_msg = "skipping byte-compilation of %s" % f self.make_file (f, out_fn, compile, (f,), compile_msg, skip_msg) # run () diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/sdist.py distutils.patched/distutils/command/sdist.py --- distutils.orig/distutils/command/sdist.py Wed Jul 5 05:06:46 2000 +++ distutils.patched/distutils/command/sdist.py Tue Jul 18 11:20:23 2000 @@ -73,7 +73,7 @@ help_options = [ ('help-formats', None, "list available distribution formats", show_formats), - ] + ] negative_opt = {'no-defaults': 'use-defaults', 'no-prune': 'prune' } diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/cygwinccompiler.py distutils.patched/distutils/cygwinccompiler.py --- distutils.orig/distutils/cygwinccompiler.py Fri Jun 30 00:57:55 2000 +++ distutils.patched/distutils/cygwinccompiler.py Tue Jul 18 11:20:23 2000 @@ -6,53 +6,45 @@ cygwin in no-cygwin mode). """ +# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a +# +# see also http://starship.python.net/crew/kernr/mingw32/Notes.html +# +# * cygwin gcc 2.91.57/ld 2.9.4/dllwrap 0.2.4 works +# (after patching python's config.h and for C++ some other include files) +# see also http://starship.python.net/crew/kernr/mingw32/Notes.html +# * mingw32 gcc 2.95.2/ld 2.9.4/dllwrap 0.2.4 works +# (ld doesn't support -shared, so we use dllwrap) +# * cygwin gcc 2.95.2/ld 2.10.90/dllwrap 2.10.90 doesn't work +# - its dllwrap doesn't work, there is a bug in binutils 2.10.90 +# see also http://sources.redhat.com/ml/cygwin/2000-06/msg01274.html +# - using gcc -shared instead dllwrap didn't work because some problems +# with the import library, maybe it needs some additional parameters? +# ( "undefined reference to '_imp__PyErr_SetString'", +# dlltool builds import libraries with '__imp__PyErr_SetString' +# and '___imp_PyErr_SetString' ) + # created 2000/05/05, Rene Liebscher __revision__ = "$Id: cygwinccompiler.py,v 1.2 2000/06/29 22:57:55 gward Exp $" -import os,sys,string -from distutils import sysconfig +import os,sys from distutils.unixccompiler import UnixCCompiler -# Because these compilers aren't configured in Python's config.h file by -# default we should at least warn the user if he is using a unmodified -# version. - -def check_config_h(): - """Checks if the GCC compiler is mentioned in config.h. If it is not, - compiling probably doesn't work, so print a warning to stderr. - """ - - # XXX the result of the check should be returned! - - from distutils import sysconfig - import string,sys - try: - # It would probably better to read single lines to search. - # But we do this only once, and it is fast enough - f=open(sysconfig.get_config_h_filename()) - s=f.read() - f.close() - try: - # is somewhere a #ifdef __GNUC__ or something similar - string.index(s,"__GNUC__") - except ValueError: - sys.stderr.write ("warning: "+ - "Python's config.h doesn't seem to support your compiler.\n") - except IOError: - # if we can't read this file, we cannot say it is wrong - # the compiler will complain later about this file as missing - pass - - -# This is called when the module is imported, so we make this check only once -# XXX why not make it only when the compiler is needed? -check_config_h() - - class CygwinCCompiler (UnixCCompiler): compiler_type = 'cygwin' + gcc_version = None + dllwrap_version = None + ld_version = None def __init__ (self, verbose=0, @@ -61,22 +53,41 @@ UnixCCompiler.__init__ (self, verbose, dry_run, force) + if check_config_h()<=0: + self.warn( + "Python's config.h doesn't seem to support your compiler. " + "Compiling may fail because of undefined preprocessor macros.") + + self.gcc_version, self.ld_version, self.dllwrap_version = get_versions() + sys.stderr.write(self.compiler_type + ": gcc %s, ld %s, dllwrap %s\n" % + (self.gcc_version, + self.ld_version, + self.dllwrap_version) ) + + # ld_version >= "2.10.90" should also be able to use + # gcc -shared instead of dllwrap + # Older dllwraps had own version numbers, newer ones use the + # same as the rest of binutils ( also ld ) + if self.dllwrap_version == "2.10.90": + # only warn, the user could have a patched version + self.warn("dllwrap 2.10.90 is buggy please use another version.") + # Hard-code GCC because that's what this is all about. # XXX optimization, warnings etc. should be customizable. - self.set_executables(compiler='gcc -O -Wall', - compiler_so='gcc -O -Wall', - linker_exe='gcc', - linker_so='dllwrap --target=i386-cygwin32') + self.set_executables(compiler='gcc -mcygwin -O -Wall', + compiler_so='gcc -mcygwin -mdll -O -Wall', + linker_exe='gcc -mcygwin', + linker_so='dllwrap -mcygwin -mdll') # cygwin and mingw32 need different sets of libraries - self.dll_libraries=[ - # cygwin shouldn't need msvcrt, - # but without the dll's will crash - # ( gcc version 2.91.57 ) - # perhaps something about initialization - # mingw32 needs it in all cases - "msvcrt" - ] + if self.gcc_version == "2.91.57": + # cygwin shouldn't need msvcrt, but without the dlls will crash + # ( gcc version 2.91.57 ) perhaps something about initialization + self.dll_libraries=["msvcrt"] + self.warn( + "You should think over switching to a newer version of gcc") + else: + self.dll_libraries=[] # __init__ () @@ -96,53 +107,49 @@ if libraries == None: libraries = [] - # Additional libraries: the python library is always needed on - # Windows we need the python version without the dot, eg. '15' - - pythonlib = ("python%d%d" % - (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) - libraries.append(pythonlib) - libraries.extend(self.dll_libraries) - - # name of extension + # Additional libraries + # libraries.extend(...) would affect the list we got as parameter + # and this can come from anywhere + libraries = libraries + self.dll_libraries + + # we want to put some files in the same directory as the + # object files are, build_temp doesn't help much - # XXX WRONG WRONG WRONG - # this is NOT the place to make guesses about Python namespaces; - # that MUST be done in build_ext.py + # name of dll to give the helper files (def, lib, exp) the same name + dll_name,dll_extension = os.path.splitext(os.path.basename(output_filename)) - if not debug: - ext_name = os.path.basename(output_filename)[:-len(".pyd")] - else: - ext_name = os.path.basename(output_filename)[:-len("_d.pyd")] + # where are the object files + temp_dir = os.path.dirname(objects[0]) - def_file = os.path.join(build_temp, ext_name + ".def") - #exp_file = os.path.join(build_temp, ext_name + ".exp") - #lib_file = os.path.join(build_temp, 'lib' + ext_name + ".a") + # generate the filenames for these files + def_file = None # this will be done later, if necessary + exp_file = os.path.join(temp_dir, dll_name + ".exp") + lib_file = os.path.join(temp_dir, 'lib' + dll_name + ".a") - # Make .def file - # (It would probably better to check if we really need this, - # but for this we had to insert some unchanged parts of - # UnixCCompiler, and this is not what we want.) - f = open(def_file,"w") - f.write("EXPORTS\n") # intro - if export_symbols == None: - # export a function "init" + ext_name - f.write("init" + ext_name + "\n") - else: - # if there are more symbols to export write them into f + # check what we got in export_symbols + if export_symbols is not None: + # Make .def file + # (It would probably better to check if we really need this, + # but for this we had to insert some unchanged parts of + # UnixCCompiler, and this is not what we want.) + def_file = os.path.join(temp_dir, dll_name + ".def") + f = open(def_file,"w") + f.write("EXPORTS\n") # intro for sym in export_symbols: f.write(sym+"\n") - f.close() - - if extra_preargs == None: + f.close() + + + if extra_preargs is None: extra_preargs = [] extra_preargs = extra_preargs + [ #"--verbose", #"--output-exp",exp_file, #"--output-lib",lib_file, - "--def",def_file ] + if def_file: + extra_preargs.extend(["--def",def_file]) # who wants symbols and a many times larger output file # should explicitely switch the debug mode on @@ -150,7 +157,7 @@ # (On my machine unstripped_file = stripped_file + 254KB # 10KB < stripped_file < ??100KB ) if not debug: - extra_preargs = extra_preargs + ["-s"] + extra_preargs.append("-s") UnixCCompiler.link_shared_object(self, objects, @@ -181,19 +188,112 @@ force=0): CygwinCCompiler.__init__ (self, verbose, dry_run, force) - + + # a real mingw32 doesn't need to specify a different entry point + # but cygwin 2.91.57 in no-cygwin-mode needs it + if self.gcc_version <= "2.91.57": + entry_point = ' --entry _DllMain@12' + else: + entry_point = '' self.set_executables(compiler='gcc -mno-cygwin -O -Wall', - compiler_so='gcc -mno-cygwin -O -Wall', + compiler_so='gcc -mno-cygwin -mdll -O -Wall', linker_exe='gcc -mno-cygwin', - linker_so='dllwrap' - + ' --target=i386-mingw32' - + ' --entry _DllMain@12') - # mingw32 doesn't really need 'target' and cygwin too (it seems, - # it is enough to specify a different entry point) - - # no additional libraries need - # (only msvcrt, which is already added by CygwinCCompiler) - + linker_so='dllwrap -mno-cygwin -mdll' + + entry_point) + # Maybe we should also append -mthreads, but then the finished + # dlls need another dll (mingwm10.dll see Mingw32 docs) + # (-mthreads: Support thread-safe exception handling on `Mingw32') + + # no additional libraries needed + self.dll_libraries=[] + # __init__ () - + # class Mingw32CCompiler + +# Because these compilers aren't configured in Python's config.h file by +# default, we should at least warn the user if he is using a unmodified +# version. + +def check_config_h(): + """Checks if the GCC compiler is mentioned in config.h. If it is not, + compiling probably doesn't work. + """ + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK + + from distutils import sysconfig + import string,sys + # if sys.version contains GCC then python was compiled with + # GCC, and the config.h file should be OK + if -1 == string.find(sys.version,"GCC"): + pass # go to the next test + else: + return 2 + + try: + # It would probably better to read single lines to search. + # But we do this only once, and it is fast enough + f=open(sysconfig.get_config_h_filename()) + s=f.read() + f.close() + + # is somewhere a #ifdef __GNUC__ or something similar + if -1 == string.find(s,"__GNUC__"): + return -1 + else: + return 1 + except IOError: + # if we can't read this file, we cannot say it is wrong + # the compiler will complain later about this file as missing + pass + return 0 + +def get_versions(): + from distutils.version import StrictVersion + from distutils.spawn import find_executable + import re + + gcc_exe = find_executable('gcc') + if gcc_exe: + out = os.popen(gcc_exe + ' -dumpversion','r') + out_string = out.read() + out.close() + result = re.search('(\d+\.\d+\.\d+)',out_string) + if result: + gcc_version = StrictVersion(result.group(1)) + else: + gcc_version = None + else: + gcc_version = None + ld_exe = find_executable('ld') + if ld_exe: + out = os.popen(ld_exe + ' -v','r') + out_string = out.read() + out.close() + result = re.search('(\d+\.\d+\.\d+)',out_string) + if result: + ld_version = StrictVersion(result.group(1)) + else: + ld_version = None + else: + ld_version = None + dllwrap_exe = find_executable('dllwrap') + if dllwrap_exe: + out = os.popen(dllwrap_exe + ' --version','r') + out_string = out.read() + out.close() + result = re.search(' (\d+\.\d+\.\d+)',out_string) + if result: + dllwrap_version = StrictVersion(result.group(1)) + else: + dllwrap_version = None + else: + dllwrap_version = None + return (gcc_version, ld_version, dllwrap_version) + + diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/dist.py distutils.patched/distutils/dist.py --- distutils.orig/distutils/dist.py Fri Jul 7 22:45:21 2000 +++ distutils.patched/distutils/dist.py Tue Jul 18 11:20:23 2000 @@ -278,7 +278,7 @@ user_filename = ".pydistutils.cfg" else: user_filename = "pydistutils.cfg" - + # And look for the user config file if os.environ.has_key('HOME'): user_file = os.path.join(os.environ.get('HOME'), user_filename) @@ -437,8 +437,8 @@ negative_opt = copy (negative_opt) negative_opt.update (cmd_class.negative_opt) - # Check for help_options in command class. They have a different - # format (tuple of four) so we need to preprocess them here. + # Check for help_options in command class. They have a different + # format (tuple of four) so we need to preprocess them here. if (hasattr(cmd_class, 'help_options') and type (cmd_class.help_options) is ListType): help_options = fix_help_options(cmd_class.help_options) @@ -463,7 +463,7 @@ for (help_option, short, desc, func) in cmd_class.help_options: if hasattr(opts, parser.get_attr_name(help_option)): help_option_found=1 - #print "showing help for option %s of command %s" % \ + #print "showing help for option %s of command %s" % \ # (help_option[0],cmd_class) if callable(func): @@ -539,7 +539,7 @@ return # _show_help () - + def handle_display_options (self, option_order): """If there were any non-global "display-only" options diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/extension.py distutils.patched/distutils/extension.py --- distutils.orig/distutils/extension.py Sat Jun 24 02:18:24 2000 +++ distutils.patched/distutils/extension.py Tue Jul 18 11:20:23 2000 @@ -111,7 +111,7 @@ self.extra_objects = extra_objects or [] self.extra_compile_args = extra_compile_args or [] self.extra_link_args = extra_link_args or [] - self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols self.export_symbol_file = export_symbol_file # class Extension diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/msvccompiler.py distutils.patched/distutils/msvccompiler.py --- distutils.orig/distutils/msvccompiler.py Fri Jun 30 21:37:59 2000 +++ distutils.patched/distutils/msvccompiler.py Tue Jul 18 11:20:23 2000 @@ -379,6 +380,17 @@ ld_args = (ldflags + lib_opts + export_opts + objects + ['/OUT:' + output_filename]) + + # The MSVC linker generates .lib and .exp files, which cannot be + # suppressed by any linker switches. The .lib files may even be + # needed! Make sure they are generated in the temporary build + # directory. Since they have different names for debug and release + # builds, they can go into the same directory. + dll_name, dll_ext = os.path.splitext(os.path.basename(output_filename)) + implib_file = os.path.join ( + os.path.dirname(objects[0]), + self.static_lib_format % (dll_name, self.static_lib_extension)) + ld_args.append ('/IMPLIB:' + implib_file) if extra_preargs: ld_args[:0] = extra_preargs diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/spawn.py distutils.patched/distutils/spawn.py --- distutils.orig/distutils/spawn.py Sun Mar 26 23:47:00 2000 +++ distutils.patched/distutils/spawn.py Tue Jul 18 11:20:23 2000 @@ -1,7 +1,9 @@ """distutils.spawn Provides the 'spawn()' function, a front-end to various platform- -specific functions for launching another program in a sub-process.""" +specific functions for launching another program in a sub-process. +Also provides the 'find_executable()' to search the path for a given +executable name. """ # created 1999/07/24, Greg Ward @@ -65,17 +67,8 @@ executable = cmd[0] cmd = _nt_quote_args (cmd) if search_path: - paths = string.split( os.environ['PATH'], os.pathsep) - base,ext = os.path.splitext(executable) - if (ext != '.exe'): - executable = executable + '.exe' - if not os.path.isfile(executable): - paths.reverse() # go over the paths and keep the last one - for p in paths: - f = os.path.join( p, executable ) - if os.path.isfile ( f ): - # the file exists, we have a shot at spawn working - executable = f + # either we find one or it stays the same + executable = find_executable(executable) or executable if verbose: print string.join ([executable] + cmd[1:], ' ') if not dry_run: @@ -91,7 +84,6 @@ raise DistutilsExecError, \ "command '%s' failed with exit status %d" % (cmd[0], rc) - def _spawn_posix (cmd, search_path=1, @@ -147,3 +139,24 @@ "unknown error executing '%s': termination status %d" % \ (cmd[0], status) # _spawn_posix () + + +def find_executable(executable): + """ Tries to find executable in the directories listed in PATH. + Returns the complete filename or None if not found. """ + + paths = string.split( os.environ['PATH'], os.pathsep) + base,ext = os.path.splitext(executable) + if (sys.platform == 'win32') and (ext != '.exe'): + executable = executable + '.exe' + if not os.path.isfile(executable): + for p in paths: + f = os.path.join( p, executable ) + if os.path.isfile ( f ): + # the file exists, we have a shot at spawn working + return f + return None + else: + return executable + +# find_executable() diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/sysconfig.py distutils.patched/distutils/sysconfig.py --- distutils.orig/distutils/sysconfig.py Tue Jun 27 03:59:43 2000 +++ distutils.patched/distutils/sysconfig.py Tue Jul 18 11:20:24 2000 @@ -267,24 +267,8 @@ g['INCLUDEPY'] = get_python_inc(plat_specific=0) g['SO'] = '.pyd' - g['exec_prefix'] = EXEC_PREFIX - - # These are needed for the CygwinCCompiler and Mingw32CCompiler - # classes, which are just UnixCCompiler classes that happen to work on - # Windows. UnixCCompiler expects to find these values in sysconfig, so - # here they are. The fact that other Windows compilers don't need - # these values is pure luck (hmmm). - - # XXX I think these are now unnecessary... - - g['CC'] = "cc" # not gcc? - g['RANLIB'] = "ranlib" - g['AR'] = "ar" - g['OPT'] = "-O2" - g['SO'] = ".pyd" - g['LDSHARED'] = "ld" - g['CCSHARED'] = "" g['EXE'] = ".exe" + g['exec_prefix'] = EXEC_PREFIX def _init_mac():
On 23 July 2000, Rene Liebscher said:
In the latest cygwin binutils 2.10.90 is a bug in dllwrap. This new version of my patch tries now to find out the version numbers of gcc, ld and dllwrap, so we can inform the user if he is using a broken version of one of these programs.
Ugh. I guess version-checking is slightly better than crashing, but I hate that it's necessary. *grumble*
(I also found many tabs in distutils which were introduced by by my help-option patch. This patch replaces these tabs with spaces.)
This patch is getting too big for me to digest, and there are too many unrelated changes. Could you drop the tab fix (I'll do it myself -- been meaning to for a while now), and submit your fix to bcppcompiler.py separately from the other patches? *Then* I'll be able to comment sensibly on it. Thanks -- Greg -- Greg Ward - Linux geek gward@python.net http://starship.python.net/~gward/ An elephant: a mouse built to government specifications.
Greg Ward wrote:
On 23 July 2000, Rene Liebscher said:
In the latest cygwin binutils 2.10.90 is a bug in dllwrap. This new version of my patch tries now to find out the version numbers of gcc, ld and dllwrap, so we can inform the user if he is using a broken version of one of these programs.
Ugh. I guess version-checking is slightly better than crashing, but I hate that it's necessary. *grumble*
There is no other way (except to have a own compiler class for every version.) In the latest version of cygwinccompiler.py all cygwin/mingw compilers I could test are now working.
....., and submit your fix to bcppcompiler.py separately from the other patches? *Then* I'll be able to comment sensibly on it.
OK, here are now three patches: * msvc.patch: changes msvccompiler.py and build_ext.py (You need its changes in build_ext.py to get the other two compilers work properly after applying their patches.) * bcpp.patch: changes bcppcompiler.py * cygwin.patch: the latest version of cygwinccompiler.py, some changes in spawn.py (I moved a port of spawn_nt() in a separate function find_executable(), I use this in my compiler class) and it removes from sysconfig.py g['CC'] = "cc" and this other stuff which UnixCCompiler needed in its old version. And some cosmetic changes: Almost all messages start with a verb, but this one in install_lib.py starts with a noun. ####################### --- distutils.orig/distutils/command/install_lib.py +++ distutils.patched/distutils/command/install_lib.py @@ -74,7 +74,7 @@ out_fn = f + (__debug__ and "c" or "o") compile_msg = "byte-compiling %s to %s" % \ (f, os.path.basename (out_fn)) - skip_msg = "byte-compilation of %s skipped" % f + skip_msg = "skipping byte-compilation of %s" % f self.make_file (f, out_fn, compile, (f,), compile_msg, skip_msg) # run () ####################### Kind regards Rene Liebscher
Oops, I forgot to attach the patches. Here they are. Kind regards Rene Liebscher diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py d.px/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Thu Jul 27 09:52:46 2000 +++ d.px/distutils/command/build_ext.py Thu Jul 27 12:31:44 2000 @@ -84,7 +84,7 @@ help_options = [ ('help-compiler', None, "list available compilers", show_compilers), - ] + ] def initialize_options (self): self.extensions = None @@ -283,6 +283,9 @@ # Medium-easy stuff: same syntax/semantics, different names. ext.runtime_library_dirs = build_info.get('rpath') ext.export_symbol_file = build_info.get('def_file') + if ext.export_symbol_file is not None: + self.warn("export_symbol_file is no longer supported, " + "comments and questions to <distutils-sig@python.org>") # Non-trivial stuff: 'macros' split into 'define_macros' # and 'undef_macros'. @@ -420,16 +423,14 @@ objects.extend (ext.extra_objects) extra_args = ext.extra_link_args or [] - # Bunch of fixing-up we have to do for Microsoft's linker. - if self.compiler.compiler_type == 'msvc': - self.msvc_prelink_hack(sources, ext, extra_args) self.compiler.link_shared_object ( objects, ext_filename, - libraries=ext.libraries, + libraries=self.get_libraries(ext), library_dirs=ext.library_dirs, runtime_library_dirs=ext.runtime_library_dirs, extra_postargs=extra_args, + export_symbols=self.get_export_symbols(ext), debug=self.debug, build_temp=self.build_temp) @@ -511,44 +512,6 @@ # find_swig () - - # -- Hooks 'n hacks ------------------------------------------------ - - def msvc_prelink_hack (self, sources, ext, extra_args): - - # XXX this is a kludge! Knowledge of specific compilers or - # platforms really doesn't belong here; in an ideal world, the - # CCompiler interface would provide access to everything in a - # compiler/linker system needs to build Python extensions, and - # we would just do everything nicely and cleanly through that - # interface. However, this is a not an ideal world and the - # CCompiler interface doesn't handle absolutely everything. - # Thus, kludges like this slip in occasionally. (This is no - # excuse for committing more platform- and compiler-specific - # kludges; they are to be avoided if possible!) - - def_file = ext.export_symbol_file - - if def_file is not None: - extra_args.append ('/DEF:' + def_file) - else: - modname = string.split (ext.name, '.')[-1] - extra_args.append('/export:init%s' % modname) - - # The MSVC linker generates .lib and .exp files, which cannot be - # suppressed by any linker switches. The .lib files may even be - # needed! Make sure they are generated in the temporary build - # directory. Since they have different names for debug and release - # builds, they can go into the same directory. - implib_file = os.path.join ( - self.implib_dir, - self.get_ext_libname (ext.name)) - extra_args.append ('/IMPLIB:' + implib_file) - self.mkpath (os.path.dirname (implib_file)) - - # msvc_prelink_hack () - - # -- Name generators ----------------------------------------------- # (extension names, filenames, whatever) @@ -578,5 +541,33 @@ if os.name == 'nt' and self.debug: return apply (os.path.join, ext_path) + '_d.lib' return apply (os.path.join, ext_path) + '.lib' + + + def get_export_symbols (self, ext): + # For all platforms and compiler doesn't export all symbols + # by default, we have to specify the module's init function + # if the user doesn't provide any export information. + # All other platforms and compilers should ignore such + # export information. + if ext.export_symbols is None: + # if no exports are specified we can't build a working + # python module. + # so let us export the module's init function + return ["init" + string.split(ext.name,'.')[-1]] + else: + return ext.export_symbols + + def get_libraries (self, ext): + # The python library is always needed on Windows. + # We need the python version without the dot, eg. '15' + # MSVC doesn't really need this, because the pragma in config.h + if sys.platform == "win32": + pythonlib = ("python%d%d" % + (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) + # don't extend ext.libraries, it may be shared with other + # extensions, it is a reference to the original list + return ext.libraries + [pythonlib] + else: + return ext.libraries # class build_ext diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/extension.py d.px/distutils/extension.py --- distutils.orig/distutils/extension.py Sat Jun 24 02:18:24 2000 +++ d.px/distutils/extension.py Thu Jul 27 12:31:44 2000 @@ -111,7 +111,7 @@ self.extra_objects = extra_objects or [] self.extra_compile_args = extra_compile_args or [] self.extra_link_args = extra_link_args or [] - self.export_symbols = export_symbols or [] + self.export_symbols = export_symbols self.export_symbol_file = export_symbol_file # class Extension diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/msvccompiler.py d.px/distutils/msvccompiler.py --- distutils.orig/distutils/msvccompiler.py Fri Jun 30 21:37:59 2000 +++ d.px/distutils/msvccompiler.py Thu Jul 27 12:31:44 2000 @@ -380,6 +380,17 @@ ld_args = (ldflags + lib_opts + export_opts + objects + ['/OUT:' + output_filename]) + # The MSVC linker generates .lib and .exp files, which cannot be + # suppressed by any linker switches. The .lib files may even be + # needed! Make sure they are generated in the temporary build + # directory. Since they have different names for debug and release + # builds, they can go into the same directory. + dll_name, dll_ext = os.path.splitext(os.path.basename(output_filename)) + implib_file = os.path.join ( + os.path.dirname(objects[0]), + self.static_lib_format % (dll_name, self.static_lib_extension)) + ld_args.append ('/IMPLIB:' + implib_file) + if extra_preargs: ld_args[:0] = extra_preargs if extra_postargs: diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/bcppcompiler.py d.px/distutils/bcppcompiler.py --- distutils.orig/distutils/bcppcompiler.py Wed Jun 28 03:20:35 2000 +++ d.px/distutils/bcppcompiler.py Thu Jul 27 12:35:21 2000 @@ -11,17 +11,6 @@ # someone should sit down and factor out the common code as # WindowsCCompiler! --GPW -# XXX Lyle reports that this doesn't quite work yet: -# """...but this is what I've got so far. The compile step works fine but -# when it runs the link step I get an "out of memory" failure. Since -# spawn() echoes the command it's trying to spawn, I can type the link line -# verbatim at the DOS prompt and it links the Windows DLL correctly -- so -# the syntax is correct. There's just some weird interaction going on when -# it tries to "spawn" the link process from within the setup.py script. I'm -# not really sure how to debug this one right off-hand; obviously there's -# nothing wrong with the "spawn()" function since it's working properly for -# the compile stage.""" - __revision__ = "$Id: bcppcompiler.py,v 1.1 2000/06/28 01:20:35 gward Exp $" @@ -123,6 +112,9 @@ elif ext in self._cpp_extensions: input_opt = "-P" + src = os.path.normpath(src) + obj = os.path.normpath(obj) + output_opt = "-o" + obj self.mkpath (os.path.dirname (obj)) @@ -230,24 +222,56 @@ else: ldflags = self.ldflags_shared + # Borland C++ has problems with '/' in paths + objects = map(os.path.normpath,objects) + startup_obj = 'c0d32' - libraries.append ('mypylib') + # either exchange python15.lib in the python libs directory against + # a Borland-like one, or create one with name bcpp_python15.lib + # there and remove the pragmas from config.h + #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt') # Create a temporary exports file for use by the linker head, tail = os.path.split (output_filename) modname, ext = os.path.splitext (tail) - def_file = os.path.join (build_temp, '%s.def' % modname) + temp_dir = os.path.dirname(objects[0]) # preserve tree structure + def_file = os.path.join (temp_dir, '%s.def' % modname) f = open (def_file, 'w') f.write ('EXPORTS\n') for sym in (export_symbols or []): f.write (' %s=_%s\n' % (sym, sym)) + f.close() - ld_args = ldflags + [startup_obj] + objects + \ - [',%s,,' % output_filename] + \ - libraries + [',' + def_file] + # start building command line + # flags and options + ld_args = list(ldflags) # don't use a reference, use a separate copy + for l in library_dirs: + ld_args.append("/L%s" % os.path.normpath(l)) + # objects list + ld_args.extend([startup_obj]) + ld_args.extend(objects) + # name of dll file + ld_args.extend([',',output_filename]) + # no map file + ld_args.extend([',']) + # start libraries + ld_args.extend([',']) + for lib in libraries: + # see if we find it and if there is a bcpp specific lib + # (bcpp_xxx.lib) + l = self.find_library_file(library_dirs,lib,debug) + if l == None: + ld_args.append(lib) + # probably an bcpp internal library + # self.warn('library %s not found.' % lib) + else: + # full name which prefers bcpp_xxx.lib over xxx.lib + ld_args.append(l) + # def file for export symbols + ld_args.extend([',',def_file]) if extra_preargs: ld_args[:0] = extra_preargs @@ -325,15 +349,28 @@ def runtime_library_dir_option (self, dir): raise DistutilsPlatformError, \ - "don't know how to set runtime library search path for MSVC++" + "don't know how to set runtime library search path for Borland C++" def library_option (self, lib): return self.library_filename (lib) - def find_library_file (self, dirs, lib): - + def find_library_file (self, dirs, lib, debug=0): + # find library file + # bcpp_xxx.lib is better than xxx.lib + # and xxx_d.lib is better than xxx.lib if debug is set for dir in dirs: + if debug: + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib + "_d")) + if os.path.exists (libfile): + return libfile + libfile = os.path.join (dir, self.library_filename ("bcpp_" + lib)) + if os.path.exists (libfile): + return libfile + if debug: + libfile = os.path.join (dir, self.library_filename (lib + '_d')) + if os.path.exists (libfile): + return libfile libfile = os.path.join (dir, self.library_filename (lib)) if os.path.exists (libfile): return libfile diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/cygwinccompiler.py d.px/distutils/cygwinccompiler.py --- distutils.orig/distutils/cygwinccompiler.py Thu Jul 27 09:52:42 2000 +++ d.px/distutils/cygwinccompiler.py Thu Jul 27 12:19:55 2000 @@ -6,53 +6,53 @@ cygwin in no-cygwin mode). """ +# problems: +# +# * if you use a msvc compiled python version (1.5.2) +# 1. you have to insert a __GNUC__ section in its config.h +# 2. you have to generate a import library for its dll +# - create a def-file for python??.dll +# - create a import library using +# dlltool --dllname python15.dll --def python15.def \ +# --output-lib libpython15.a +# +# see also http://starship.python.net/crew/kernr/mingw32/Notes.html +# +# * We use put export_symbols in a def-file, and don't use +# --export-all-symbols because it doesn't worked reliable in some +# tested configurations. And because other windows compilers also +# need their symbols specified this no serious problem. +# +# tested configurations: +# +# * cygwin gcc 2.91.57/ld 2.9.4/dllwrap 0.2.4 works +# (after patching python's config.h and for C++ some other include files) +# see also http://starship.python.net/crew/kernr/mingw32/Notes.html +# * mingw32 gcc 2.95.2/ld 2.9.4/dllwrap 0.2.4 works +# (ld doesn't support -shared, so we use dllwrap) +# * cygwin gcc 2.95.2/ld 2.10.90/dllwrap 2.10.90 works now +# - its dllwrap doesn't work, there is a bug in binutils 2.10.90 +# see also ..... +# - using gcc -mdll instead dllwrap doesn't work without -static because +# it tries to link against dlls instead their import libraries. (If +# it finds the dll first.) +# By specifying -static we force ld to link against the import libraries, +# this is windows standard and there are normally not the necessary symbols +# in the dlls. + # created 2000/05/05, Rene Liebscher __revision__ = "$Id: cygwinccompiler.py,v 1.3 2000/07/27 02:13:19 gward Exp $" -import os,sys,string -from distutils import sysconfig +import os,sys from distutils.unixccompiler import UnixCCompiler -# Because these compilers aren't configured in Python's config.h file by -# default we should at least warn the user if he is using a unmodified -# version. - -def check_config_h(): - """Checks if the GCC compiler is mentioned in config.h. If it is not, - compiling probably doesn't work, so print a warning to stderr. - """ - - # XXX the result of the check should be returned! - - from distutils import sysconfig - import string,sys - try: - # It would probably better to read single lines to search. - # But we do this only once, and it is fast enough - f=open(sysconfig.get_config_h_filename()) - s=f.read() - f.close() - try: - # is somewhere a #ifdef __GNUC__ or something similar - string.index(s,"__GNUC__") - except ValueError: - sys.stderr.write ("warning: "+ - "Python's config.h doesn't seem to support your compiler.\n") - except IOError: - # if we can't read this file, we cannot say it is wrong - # the compiler will complain later about this file as missing - pass - - -# This is called when the module is imported, so we make this check only once -# XXX why not make it only when the compiler is needed? -check_config_h() - - class CygwinCCompiler (UnixCCompiler): compiler_type = 'cygwin' + gcc_version = None + dllwrap_version = None + ld_version = None def __init__ (self, verbose=0, @@ -61,22 +61,43 @@ UnixCCompiler.__init__ (self, verbose, dry_run, force) + if check_config_h()<=0: + self.warn( + "Python's config.h doesn't seem to support your compiler. " + "Compiling may fail because of undefined preprocessor macros.") + + self.gcc_version, self.ld_version, self.dllwrap_version = get_versions() + sys.stderr.write(self.compiler_type + ": gcc %s, ld %s, dllwrap %s\n" % + (self.gcc_version, + self.ld_version, + self.dllwrap_version) ) + + # ld_version >= "2.10.90" should also be able to use + # gcc -mdll instead of dllwrap + # Older dllwraps had own version numbers, newer ones use the + # same as the rest of binutils ( also ld ) + # dllwrap 2.10.90 is buggy + if self.ld_version >= "2.10.90": + self.linker = "gcc" + else: + self.linker = "dllwrap" + # Hard-code GCC because that's what this is all about. # XXX optimization, warnings etc. should be customizable. - self.set_executables(compiler='gcc -O -Wall', - compiler_so='gcc -O -Wall', - linker_exe='gcc', - linker_so='dllwrap --target=i386-cygwin32') + self.set_executables(compiler='gcc -mcygwin -O -Wall', + compiler_so='gcc -mcygwin -mdll -O -Wall', + linker_exe='gcc -mcygwin', + linker_so=('%s -mcygwin -mdll -static' % self.linker)) # cygwin and mingw32 need different sets of libraries - self.dll_libraries=[ - # cygwin shouldn't need msvcrt, - # but without the dll's will crash - # ( gcc version 2.91.57 ) - # perhaps something about initialization - # mingw32 needs it in all cases - "msvcrt" - ] + if self.gcc_version == "2.91.57": + # cygwin shouldn't need msvcrt, but without the dlls will crash + # ( gcc version 2.91.57 ) perhaps something about initialization + self.dll_libraries=["msvcrt"] + self.warn( + "You should think over switching to a newer version of gcc") + else: + self.dll_libraries=[] # __init__ () @@ -93,64 +114,65 @@ extra_postargs=None, build_temp=None): - if libraries == None: - libraries = [] + # use seperate copies, so can modify the lists + extra_preargs = list(extra_preargs or []) + libraries = list(libraries or []) - # Additional libraries: the python library is always needed on - # Windows we need the python version without the dot, eg. '15' - - pythonlib = ("python%d%d" % - (sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff)) - libraries.append(pythonlib) + # Additional libraries libraries.extend(self.dll_libraries) + + # we want to put some files in the same directory as the + # object files are, build_temp doesn't help much - # name of extension + # where are the object files + temp_dir = os.path.dirname(objects[0]) - # XXX WRONG WRONG WRONG - # this is NOT the place to make guesses about Python namespaces; - # that MUST be done in build_ext.py + # name of dll to give the helper files (def, lib, exp) the same name + dll_name,dll_extension = os.path.splitext(os.path.basename(output_filename)) - if not debug: - ext_name = os.path.basename(output_filename)[:-len(".pyd")] - else: - ext_name = os.path.basename(output_filename)[:-len("_d.pyd")] + # generate the filenames for these files + def_file = None # this will be done later, if necessary + exp_file = os.path.join(temp_dir, dll_name + ".exp") + lib_file = os.path.join(temp_dir, 'lib' + dll_name + ".a") - def_file = os.path.join(build_temp, ext_name + ".def") - #exp_file = os.path.join(build_temp, ext_name + ".exp") - #lib_file = os.path.join(build_temp, 'lib' + ext_name + ".a") - - # Make .def file - # (It would probably better to check if we really need this, - # but for this we had to insert some unchanged parts of - # UnixCCompiler, and this is not what we want.) - f = open(def_file,"w") - f.write("EXPORTS\n") # intro - if export_symbols == None: - # export a function "init" + ext_name - f.write("init" + ext_name + "\n") + #extra_preargs.append("--verbose") + if self.linker == "dllwrap": + extra_preargs.extend([#"--output-exp",exp_file, + "--output-lib",lib_file, + ]) else: - # if there are more symbols to export write them into f + # doesn't work: bfd_close build\...\libfoo.a: Invalid operation + extra_preargs.extend([#"-Wl,--out-implib,%s" % lib_file, + ]) + + # check what we got in export_symbols + if export_symbols is not None: + # Make .def file + # (It would probably better to check if we really need this, + # but for this we had to insert some unchanged parts of + # UnixCCompiler, and this is not what we want.) + def_file = os.path.join(temp_dir, dll_name + ".def") + f = open(def_file,"w") + f.write("LIBRARY %s\n" % os.path.basename(output_filename)) + f.write("EXPORTS\n") # intro for sym in export_symbols: f.write(sym+"\n") - f.close() - - if extra_preargs == None: - extra_preargs = [] - - extra_preargs = extra_preargs + [ - #"--verbose", - #"--output-exp",exp_file, - #"--output-lib",lib_file, - "--def",def_file - ] - + f.close() + + if def_file: + if self.linker == "dllwrap": + # for dllwrap we have to use a special option + extra_preargs.append("--def") + # for gcc/ld it is specified as any other object file + extra_preargs.append(def_file) + # who wants symbols and a many times larger output file # should explicitly switch the debug mode on - # otherwise we let dllwrap strip the output file + # otherwise we let dllwrap/ld strip the output file # (On my machine unstripped_file = stripped_file + 254KB # 10KB < stripped_file < ??100KB ) if not debug: - extra_preargs = extra_preargs + ["-s"] + extra_preargs.append("-s") UnixCCompiler.link_shared_object(self, objects, @@ -159,7 +181,7 @@ libraries, library_dirs, runtime_library_dirs, - None, # export_symbols, we do this with our def-file + None, # export_symbols, we do this in our def-file debug, extra_preargs, extra_postargs, @@ -181,19 +203,115 @@ force=0): CygwinCCompiler.__init__ (self, verbose, dry_run, force) + + # a real mingw32 doesn't need to specify a different entry point + # but cygwin 2.91.57 in no-cygwin-mode needs it + if self.gcc_version <= "2.91.57": + entry_point = '--entry _DllMain@12' + else: + entry_point = '' self.set_executables(compiler='gcc -mno-cygwin -O -Wall', - compiler_so='gcc -mno-cygwin -O -Wall', + compiler_so='gcc -mno-cygwin -mdll -O -Wall', linker_exe='gcc -mno-cygwin', - linker_so='dllwrap' - + ' --target=i386-mingw32' - + ' --entry _DllMain@12') - # mingw32 doesn't really need 'target' and cygwin too (it seems, - # it is enough to specify a different entry point) - - # no additional libraries need - # (only msvcrt, which is already added by CygwinCCompiler) - + linker_so='%s -mno-cygwin -mdll -static %s' + % (self.linker, entry_point)) + # Maybe we should also append -mthreads, but then the finished + # dlls need another dll (mingwm10.dll see Mingw32 docs) + # (-mthreads: Support thread-safe exception handling on `Mingw32') + + # no additional libraries needed + self.dll_libraries=[] + # __init__ () - + # class Mingw32CCompiler + +# Because these compilers aren't configured in Python's config.h file by +# default, we should at least warn the user if he is using a unmodified +# version. + +def check_config_h(): + """Checks if the GCC compiler is mentioned in config.h. If it is not, + compiling probably doesn't work. + """ + # return values + # 2: OK, python was compiled with GCC + # 1: OK, python's config.h mentions __GCC__ + # 0: uncertain, because we couldn't check it + # -1: probably not OK, because we didn't found it in config.h + # You could check check_config_h()>0 => OK + + from distutils import sysconfig + import string,sys + # if sys.version contains GCC then python was compiled with + # GCC, and the config.h file should be OK + if -1 == string.find(sys.version,"GCC"): + pass # go to the next test + else: + return 2 + + try: + # It would probably better to read single lines to search. + # But we do this only once, and it is fast enough + f=open(sysconfig.get_config_h_filename()) + s=f.read() + f.close() + + # is somewhere a #ifdef __GNUC__ or something similar + if -1 == string.find(s,"__GNUC__"): + return -1 + else: + return 1 + except IOError: + # if we can't read this file, we cannot say it is wrong + # the compiler will complain later about this file as missing + pass + return 0 + +def get_versions(): + """ Try to find out the versions of gcc, ld and dllwrap. + If not possible it returns None for it. + """ + from distutils.version import StrictVersion + from distutils.spawn import find_executable + import re + + gcc_exe = find_executable('gcc') + if gcc_exe: + out = os.popen(gcc_exe + ' -dumpversion','r') + out_string = out.read() + out.close() + result = re.search('(\d+\.\d+\.\d+)',out_string) + if result: + gcc_version = StrictVersion(result.group(1)) + else: + gcc_version = None + else: + gcc_version = None + ld_exe = find_executable('ld') + if ld_exe: + out = os.popen(ld_exe + ' -v','r') + out_string = out.read() + out.close() + result = re.search('(\d+\.\d+\.\d+)',out_string) + if result: + ld_version = StrictVersion(result.group(1)) + else: + ld_version = None + else: + ld_version = None + dllwrap_exe = find_executable('dllwrap') + if dllwrap_exe: + out = os.popen(dllwrap_exe + ' --version','r') + out_string = out.read() + out.close() + result = re.search(' (\d+\.\d+\.\d+)',out_string) + if result: + dllwrap_version = StrictVersion(result.group(1)) + else: + dllwrap_version = None + else: + dllwrap_version = None + return (gcc_version, ld_version, dllwrap_version) + diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/spawn.py d.px/distutils/spawn.py --- distutils.orig/distutils/spawn.py Sun Mar 26 23:47:00 2000 +++ d.px/distutils/spawn.py Thu Jul 27 12:29:03 2000 @@ -1,7 +1,9 @@ """distutils.spawn Provides the 'spawn()' function, a front-end to various platform- -specific functions for launching another program in a sub-process.""" +specific functions for launching another program in a sub-process. +Also provides the 'find_executable()' to search the path for a given +executable name. """ # created 1999/07/24, Greg Ward @@ -65,17 +67,8 @@ executable = cmd[0] cmd = _nt_quote_args (cmd) if search_path: - paths = string.split( os.environ['PATH'], os.pathsep) - base,ext = os.path.splitext(executable) - if (ext != '.exe'): - executable = executable + '.exe' - if not os.path.isfile(executable): - paths.reverse() # go over the paths and keep the last one - for p in paths: - f = os.path.join( p, executable ) - if os.path.isfile ( f ): - # the file exists, we have a shot at spawn working - executable = f + # either we find one or it stays the same + executable = find_executable(executable) or executable if verbose: print string.join ([executable] + cmd[1:], ' ') if not dry_run: @@ -91,7 +84,6 @@ raise DistutilsExecError, \ "command '%s' failed with exit status %d" % (cmd[0], rc) - def _spawn_posix (cmd, search_path=1, @@ -147,3 +139,24 @@ "unknown error executing '%s': termination status %d" % \ (cmd[0], status) # _spawn_posix () + + +def find_executable(executable): + """ Tries to find executable in the directories listed in PATH. + Returns the complete filename or None if not found. """ + + paths = string.split( os.environ['PATH'], os.pathsep) + base,ext = os.path.splitext(executable) + if (sys.platform == 'win32') and (ext != '.exe'): + executable = executable + '.exe' + if not os.path.isfile(executable): + for p in paths: + f = os.path.join( p, executable ) + if os.path.isfile ( f ): + # the file exists, we have a shot at spawn working + return f + return None + else: + return executable + +# find_executable() diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/sysconfig.py d.px/distutils/sysconfig.py --- distutils.orig/distutils/sysconfig.py Tue Jun 27 03:59:43 2000 +++ d.px/distutils/sysconfig.py Thu Jul 27 12:29:03 2000 @@ -267,24 +267,8 @@ g['INCLUDEPY'] = get_python_inc(plat_specific=0) g['SO'] = '.pyd' - g['exec_prefix'] = EXEC_PREFIX - - # These are needed for the CygwinCCompiler and Mingw32CCompiler - # classes, which are just UnixCCompiler classes that happen to work on - # Windows. UnixCCompiler expects to find these values in sysconfig, so - # here they are. The fact that other Windows compilers don't need - # these values is pure luck (hmmm). - - # XXX I think these are now unnecessary... - - g['CC'] = "cc" # not gcc? - g['RANLIB'] = "ranlib" - g['AR'] = "ar" - g['OPT'] = "-O2" - g['SO'] = ".pyd" - g['LDSHARED'] = "ld" - g['CCSHARED'] = "" g['EXE'] = ".exe" + g['exec_prefix'] = EXEC_PREFIX def _init_mac():
On 27 July 2000, Rene Liebscher said:
* msvc.patch: changes msvccompiler.py and build_ext.py (You need its changes in build_ext.py to get the other two compilers work properly after applying their patches.)
OK, I've checked that one in. One problem: I don't like the fact that Extension.ext_symbols can now be None -- you took away the "or []" in the constructor, which is inconsistent with every other attribute of Extension. Since this class is just a pile of data without much behaviour, I value convenience -- don't have to check for None, just assume everything is a list -- over completeness. And I really dislike the inconsistency you've introduced. I can see why None-ness is needed, though: in build_ext.get_export_symbols(), you can tell if 'export_symbols' was supplied at all by looking for None. But if an empty list was supplied, isn't that wrong? If export_symbols is an empty list, then we'll go ahead and build the .pyd with nothing in the export list, not even "init" + module_name. In the general-purpose library-building code (msvccompiler.py), this would be right: but this code is geared towards building Python extensions, and if a Python extension doesn't export its init function, what's the point? I suggest changing build_ext.get_export_symbols() from this: if ext.export_symbols is None: return [] else: return ext.export_symbols to this: initfunc_name = "init" + string.split(ext.name,'.')[-1] if initfunc_name not in ext.export_symbols: ext.export_symbols.append(initfunc_name) which of course assumes that ext.export_symbols is a list, which will be valid as long as we put back the "or []" in Extension's constructor. Sound reasonable?
* bcpp.patch: changes bcppcompiler.py
OK, I'm going over this one now.
# either exchange python15.lib in the python libs directory against # a Borland-like one, or create one with name bcpp_python15.lib # there and remove the pragmas from config.h #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt')
I don't understand. You commented out the use of 'mypylib' and added the comment. Can you explain what's going on here, please? If we need to make changes in Python (1.6 or 2.0!) to support Borland C++ better, now's the time to do it.
ld_args.extend([',',output_filename]) # no map file ld_args.extend([',']) # start libraries ld_args.extend([','])
This code seems fishy to me. First of all, why are you using 'extend([x])' when 'append(x)' would do the same thing? And second, are those commas really supposed to be whole words in the arg list? I don't know how things work in Windows, but in Unix doing that would mean that the compiler's argv list would look like [..., ',', 'foo.pyd', ',', ',', ...] which is really strange. I suspect that really should be coded as ld_args.append(",%s,," % output_filename) which would make an argv of [..., ',foo.pyd,,', ...] which seems a) more sensible, and b) looks like the Borland C++ command lines I remember from 7-8 years ago. So: which arguments really have to be separate words (ie. separate elements of the argv[] array that BCPP will ultimately have to process)? Does this question even make sense on a DOS/Windows environment, ie. is the argument delineation implicit in the 'spawn()' call preserved in the child process?
- def find_library_file (self, dirs, lib): - + def find_library_file (self, dirs, lib, debug=0): + # find library file
Umm, the 'debug' flag is not part of the official CCompiler interface to 'find_library_file()'. Is this really necessary? Should MSVCCompiler do it as well? If so, then the interface should change. Looks like 'find_library_file()' is used in BCPPCompiler to give preference to bcpp_xxx.lib; is this really important? Where does it arise? Is this a common convention, or a Python-only convention? (Or a Liebscher-only convention? ;-) I also note some lines of code > 80 columns in this file. Please keep all code under 75 columns!
* cygwin.patch: the latest version of cygwinccompiler.py, some changes in spawn.py (I moved a port of spawn_nt() in a separate function find_executable(), I use this in my compiler class) and it removes from sysconfig.py g['CC'] = "cc" and this other stuff which UnixCCompiler needed in its old version.
In future, please supply a checkin comment for cygwinccompiler.py -- I'll figure out this time myself, but as long as the code is your responsibility, so too are the checkin messages. Other comments... * why are {gcc,ld,dllwrap}_version class attributes? They are derived from the particular compiler used by a particular instance of the class, and when you set them in the constructor they become instance attributes. So why even define them as class attrs? * 'check_config_h()' really should return meaningful strings instead of magic numbers. Comment or no, any occurence of a bare integer in non-arithmetic circumstances is a magic number in my book, and I don't like magic numbers. * again, there are lines > 80 columns! (I've fixed most of these.) * you shouldn't write to stdout or stderr in a compiler class -- use the 'announce()' method. I should really add a 'debug_print()' method; that would be handy. * I don't like using 'list(foo)' to make a copy of a list 'foo' -- either 'foo[:]' or 'copy(foo)' (from the standard copy module) is acceptable. * when assigning to a tuple, please use parens: (a, b) = ... not a, b = ... OK, I've checked it in with a few tweaks (fixed long lines and tuple assignments, mostly). Please submit a new patch fixing the above problems.
And some cosmetic changes: Almost all messages start with a verb, but this one in install_lib.py starts with a noun. [...] - skip_msg = "byte-compilation of %s skipped" % f + skip_msg = "skipping byte-compilation of %s" % f
Thanks! At the cost of one extra character, I'll take it. That has been a very teeny tiny irritant for a while now. Greg -- Greg Ward - Linux geek gward@python.net http://starship.python.net/~gward/ I don't understand the HUMOUR of the THREE STOOGES!!
Greg Ward wrote:
On 27 July 2000, Rene Liebscher said:
* msvc.patch: changes msvccompiler.py and build_ext.py (You need its changes in build_ext.py to get the other two compilers work properly after applying their patches.)
OK, I've checked that one in. One problem: I don't like the fact that Extension.ext_symbols can now be None -- you took away the "or []" in the constructor, which is inconsistent with every other attribute of Extension. Since this class is just a pile of data without much behaviour, I value convenience -- don't have to check for None, just assume everything is a list -- over completeness. And I really dislike the inconsistency you've introduced.
I can see why None-ness is needed, though: in build_ext.get_export_symbols(), you can tell if 'export_symbols' was supplied at all by looking for None. But if an empty list was supplied, isn't that wrong? If export_symbols is an empty list, then we'll go ahead and build the .pyd with nothing in the export list, not even "init" + module_name. In the general-purpose library-building code (msvccompiler.py), this would be right: but this code is geared towards building Python extensions, and if a Python extension doesn't export its init function, what's the point?
I suggest changing build_ext.get_export_symbols() from this:
if ext.export_symbols is None: return [] else: return ext.export_symbols
to this:
initfunc_name = "init" + string.split(ext.name,'.')[-1] if initfunc_name not in ext.export_symbols: ext.export_symbols.append(initfunc_name)
which of course assumes that ext.export_symbols is a list, which will be valid as long as we put back the "or []" in Extension's constructor. Sound reasonable?
OK, I think you are right, if one wants to build none Python dll's he will not use build_ext to do this.
* bcpp.patch: changes bcppcompiler.py
OK, I'm going over this one now.
ld_args.extend([',',output_filename]) # no map file ld_args.extend([',']) # start libraries ld_args.extend([','])
This code seems fishy to me. First of all, why are you using 'extend([x])' when 'append(x)' would do the same thing? And second, are those commas really supposed to be whole words in the arg list? I don't know how things work in Windows, but in Unix doing that would mean that the compiler's argv list would look like
I used 'extend' to be able to write " ld_args.extend([',','some_other_param']) ", but I never needed it in some places and simply forgot to replace it with 'append' in the final patch.
[..., ',', 'foo.pyd', ',', ',', ...]
which is really strange. I suspect that really should be coded as
ld_args.append(",%s,," % output_filename)
which would make an argv of
[..., ',foo.pyd,,', ...]
which seems a) more sensible, and b) looks like the Borland C++ command lines I remember from 7-8 years ago.
So: which arguments really have to be separate words (ie. separate elements of the argv[] array that BCPP will ultimately have to process)? Does this question even make sense on a DOS/Windows environment, ie. is the argument delineation implicit in the 'spawn()' call preserved in the child process?
The commas separate different blocks of the command line, and I used them also to separate the command line building code in several blocks (objects, output filename, libraries, ... In the patch this block building is more obvious then in the current version where you already have put some blocks together.) It is not necessary to have these commas as separate words, but they don't really belong to a output filename or a library name, this is the other reason why I made them in separate words.
- def find_library_file (self, dirs, lib): - + def find_library_file (self, dirs, lib, debug=0): + # find library file
Umm, the 'debug' flag is not part of the official CCompiler interface to 'find_library_file()'. Is this really necessary? Should MSVCCompiler do it as well? If so, then the interface should change.
Yes, it should. Python comes also with libraries 'python??.lib' and 'python??_d.lib'. So it is probably a good idea to support it this way.
Looks like 'find_library_file()' is used in BCPPCompiler to give preference to bcpp_xxx.lib; is this really important? Where does it arise? Is this a common convention, or a Python-only convention? (Or a Liebscher-only convention? ;-) go down to the next explanation, it is about the same problem.
# either exchange python15.lib in the python libs directory against # a Borland-like one, or create one with name bcpp_python15.lib # there and remove the pragmas from config.h #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt')
I don't understand. You commented out the use of 'mypylib' and added the comment. Can you explain what's going on here, please? If we need to make changes in Python (1.6 or 2.0!) to support Borland C++ better, now's the time to do it. 'mypylib' is not a Borland standard library. It is a Borland-like python.lib. MSVC and Borland use different formats for their object files (COFF and OMF.) Now there comes also a 'python??' library from build_ext it should be also called 'python??.lib'. So you had to replace the standard 'python??.lib' by a Borland-like one. But what happens if you installed Python on a network drive and some users want to use MSVC and other BORLAND C++? Then you have to give one of these libraries another name. I would suggest 'bcpp_python??.lib'.
* conventions (see above) I have never heard of some convention how to handle libraries in the same directory if you want to use different compilers which need these libraries with the same name but different format. How to resolve this? I think it is the best if every compiler(class) first looks if there is a special version for and use then this special version. In this case bcpp_python??.lib if it exists and if not python??.lib. This would be also work with other compiler classes, for example I am also testing LCC-Win32, which uses then lcc_python??.lib . So in this sense it is a 'Liebscher-only convention', but it doesn't hurt users who replace their libraries (python??.lib, ...) with versions specific to their compilers. If you know a better way to handle this problem, let me know. * About the 'pragma' in config.h: In PC/config.h of the Python source there are following lines: ------------------------------------------- #ifndef USE_DL_EXPORT /* So nobody needs to specify the .lib in their Makefile any more */ #ifdef _DEBUG #pragma comment(lib,"python20_d.lib") #else #pragma comment(lib,"python20.lib") #endif #endif /* USE_DL_EXPORT */ ------------------------------------------- If we want to use bcpp_python20.lib instead of python20.lib, we had to change the first line to: ------------------------------------------ #if !defined(USE_DL_EXPORT) && defined(_MSC_VER) ------------------------------------------ So the Borland C++ compiler wouldn't try to load this incompatible library 'python20.lib' . It would make sense to change this in any case for the next version of Python, because it comes with an import library for MSVC, so this pragma should be visible only for MSVC. (We can't change the config.h file in existing installations this is why I put the comment in there. So if a user got problems he finds a hint where this could come from.)
* cygwin.patch: Other comments...
* why are {gcc,ld,dllwrap}_version class attributes? They are derived from the particular compiler used by a particular instance of the class, and when you set them in the constructor they become instance attributes. So why even define them as class attrs? Ok, I will change this.
* 'check_config_h()' really should return meaningful strings instead of magic numbers. Comment or no, any occurence of a bare integer in non-arithmetic circumstances is a magic number in my book, and I don't like magic numbers.
* you shouldn't write to stdout or stderr in a compiler class -- use the 'announce()' method. I should really add a 'debug_print()' method; that would be handy. If you add 'debug_print()' I will use it.
* I don't like using 'list(foo)' to make a copy of a list 'foo' -- either 'foo[:]' or 'copy(foo)' (from the standard copy module) is acceptable. Ok, I will change this too.
Kind regards Rene Liebscher
[my suggestion]
initfunc_name = "init" + string.split(ext.name,'.')[-1] if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name) [Rene agrees]
OK, I think you are right, if one wants to build none Python dll's he will not use build_ext to do this.
OK, can you make this change and include it in your next patch?
I used 'extend' to be able to write " ld_args.extend([',','some_other_param']) ", but I never needed it in some places and simply forgot to replace it with 'append' in the final patch. [...] It is not necessary to have these commas as separate words, but they don't really belong to a output filename or a library name, this is the other reason why I made them in separate words.
OK. I'd really like to see that code cleaned up: can you take care of that too? If it doesn't matter how things are broken up into words, how about just economizing on the amount of Python code? The current implementation seems rather bloated to me.
'mypylib' is not a Borland standard library. It is a Borland-like python.lib. MSVC and Borland use different formats for their object files (COFF and OMF.) Now there comes also a 'python??' library from build_ext it should be also called 'python??.lib'. So you had to replace the standard 'python??.lib' by a Borland-like one. But what happens if you installed Python on a network drive and some users want to use MSVC and other BORLAND C++? Then you have to give one of these libraries another name. I would suggest 'bcpp_python??.lib'.
OK, that seems like a sensible convention to me. Congratulations, you have just converted a "Liebscher convention" to a "Distutils convention". Next thing you know it'll be a "Python convention", and then it'll be unstoppable! ;-) I've added the 'debug' flag to the official 'find_library_file()' signature specified by CCompiler. I've also rewritten the implementation in both MSVCCompiler and BCPPCompiler. Here's the MSVC one (it's a bit simpler -- no prefix): def find_library_file (self, dirs, lib, debug=0): # Prefer a debugging library if found (and requested), but deal # with it if we don't have one. if debug: try_names = [lib + "_d", lib] else: try_names = [lib] for dir in dirs: for name in try_names: libfile = os.path.join(dir, self.library_filename (name)) if os.path.exists(libfile): return libfile else: # Oops, didn't find it in *any* of 'dirs' return None and here's the BCPP version: def find_library_file (self, dirs, lib, debug=0): # List of effective library names to try, in order of preference: # bcpp_xxx.lib is better than xxx.lib # and xxx_d.lib is better than xxx.lib if debug is set # # The "bcpp_" prefix is to handle a Python installation for people # with multiple compilers (primarily Distutils hackers, I suspect # ;-). The idea is they'd have one static library for each # compiler they care about, since (almost?) every Windows compiler # seems to have a different format for static libraries. if debug: dlib = (lib + "_d") try_names = ("bcpp_" + dlib, "bcpp_" + lib, dlib, lib) else: try_names = ("bcpp_" + lib, lib) for dir in dirs: for name in try_names: libfile = os.path.join(dir, self.library_filename (name)) if os.path.exists(libfile): return libfile else: # Oops, didn't find it in *any* of 'dirs' return None Do those look right to you? Obviously this is completely untested, but I think both are a huge improvement so I'll check 'em in anyways. Make sure you "cvs up" if you have any patches for msvccompiler.py or bcppcompiler.py! Also in the "obvious" category: these should be combined, presumably when someone (probably me, sigh) sits down and factors WindowsCCompiler out of MSVCCompiler and BCPPCompiler.
* About the 'pragma' in config.h: In PC/config.h of the Python source there are following lines: ------------------------------------------- #ifndef USE_DL_EXPORT /* So nobody needs to specify the .lib in their Makefile any more */ #ifdef _DEBUG #pragma comment(lib,"python20_d.lib") #else #pragma comment(lib,"python20.lib") #endif #endif /* USE_DL_EXPORT */ ------------------------------------------- If we want to use bcpp_python20.lib instead of python20.lib, we had to change the first line to: ------------------------------------------ #if !defined(USE_DL_EXPORT) && defined(_MSC_VER) ------------------------------------------ So the Borland C++ compiler wouldn't try to load this incompatible library 'python20.lib' .
Looks good to me. I just posted to python-dev about this: let's see what they say; I can make this change to PC/config.h if it's accepted.
(We can't change the config.h file in existing installations this is why I put the comment in there. So if a user got problems he finds a hint where this could come from.)
Sorry, which comment? Do you mean the warning that's printed if 'check_config_h()' doesn't like what it finds?
* you shouldn't write to stdout or stderr in a compiler class -- use the 'announce()' method. I should really add a 'debug_print()' method; that would be handy. If you add 'debug_print()' I will use it.
Added it. Please change your prints or writes or whatever they are. Thanks! Greg -- Greg Ward - just another Python hacker gward@python.net http://starship.python.net/~gward/ Eschew obfuscation!
Greg Ward wrote:
[my suggestion]
initfunc_name = "init" + string.split(ext.name,'.')[-1] if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name)
[Rene agrees]
OK, I think you are right, if one wants to build none Python dll's he will not use build_ext to do this.
OK, can you make this change and include it in your next patch?
I used 'extend' to be able to write " ld_args.extend([',','some_other_param']) ", but I never needed it in some places and simply forgot to replace it with 'append' in the final patch. [...] It is not necessary to have these commas as separate words, but they don't really belong to a output filename or a library name, this is the other reason why I made them in separate words.
OK. I'd really like to see that code cleaned up: can you take care of that too? If it doesn't matter how things are broken up into words, how about just economizing on the amount of Python code? The current implementation seems rather bloated to me.
It is not as easy as you may think, I just spotted a potential problem before it could arise. If I would write ...append(",,%s," % filename) and filename would contain a blank ("/a b/file") then spawn would quote it as whole ",,/a b/file,". And this is probably wrong. So we should insert file names as single words in the list. (Don't say blanks in file names are not allowed in python module names, THIS is a generic compiler class which could also be used for other purposes.)
I've added the 'debug' flag to the official 'find_library_file()' signature specified by CCompiler. I've also rewritten the implementation in both MSVCCompiler and BCPPCompiler. Here's the MSVC one (it's a bit simpler -- no prefix): ... and here's the BCPP version: ... Do those look right to you? Obviously this is completely untested, but I think both are a huge improvement so I'll check 'em in anyways. Make sure you "cvs up" if you have any patches for msvccompiler.py or bcppcompiler.py!
Also in the "obvious" category: these should be combined, presumably when someone (probably me, sigh) sits down and factors WindowsCCompiler out of MSVCCompiler and BCPPCompiler.
(We can't change the config.h file in existing installations this is why I put the comment in there. So if a user got problems he finds a hint where this could come from.)
Sorry, which comment? Do you mean the warning that's printed if 'check_config_h()' doesn't like what it finds?
I meant this one:
# either exchange python15.lib in the python libs directory against # a Borland-like one, or create one with name bcpp_python15.lib # there and remove the pragmas from config.h #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt')
* you shouldn't write to stdout or stderr in a compiler class -- use the 'announce()' method. I should really add a 'debug_print()' method; that would be handy. If you add 'debug_print()' I will use it.
Added it. Please change your prints or writes or whatever they are. Already done.
(My first email was sent uncomplete, here comes the second part.) Patch decription: - Extension.py: * ext.export_symbols is now always a list (added 'or []') - build_ext.py: * get_export_symbols() changed, adds now module init function if not given by the user - bccp_compiler.py: * changed some list.extend([...]) to list.append(...) * added '/g0' to compiler_options, so compiler doesn't stop after 100 warnings - cygwin_compiler.py: * use self.debug_print() for debug messages * uses now copy.copy() to copy lists * added 'shared_lib_extension=".dll"', ... , this is necessary if you want use the compiler class outside of the standard distutils build process. * changed result type of check_config_h() from int to string kind regards Rene Liebscher diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/bcppcompiler.py d.p/distutils/bcppcompiler.py --- distutils.orig/distutils/bcppcompiler.py Tue Aug 8 13:18:37 2000 +++ d.p/distutils/bcppcompiler.py Tue Aug 8 14:27:43 2000 @@ -67,8 +67,8 @@ self.lib = "tlib.exe" self.preprocess_options = None - self.compile_options = ['/tWM', '/O2', '/q'] - self.compile_options_debug = ['/tWM', '/Od', '/q'] + self.compile_options = ['/tWM', '/O2', '/q', '/g0'] + self.compile_options_debug = ['/tWM', '/Od', '/q', '/g0'] self.ldflags_shared = ['/Tpd', '/Gn', '/q', '/x'] self.ldflags_shared_debug = ['/Tpd', '/Gn', '/q', '/x'] @@ -232,7 +232,6 @@ # either exchange python15.lib in the python libs directory against # a Borland-like one, or create one with name bcpp_python15.lib # there and remove the pragmas from config.h - #libraries.append ('mypylib') libraries.append ('import32') libraries.append ('cw32mt') @@ -257,7 +256,7 @@ # name of dll file ld_args.extend([',',output_filename]) # no map file and start libraries - ld_args.extend([',', ',']) + ld_args.append(',,') for lib in libraries: # see if we find it and if there is a bcpp specific lib diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py d.p/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Thu Aug 3 11:00:01 2000 +++ d.p/distutils/command/build_ext.py Tue Aug 8 14:26:06 2000 @@ -549,14 +549,10 @@ the .pyd file (DLL) must export the module "init" function. """ - # XXX what if 'export_symbols' defined but it doesn't contain - # "init" + module_name? Should we add it? warn? or just carry - # on doing nothing? - - if ext.export_symbols is None: - return ["init" + string.split(ext.name,'.')[-1]] - else: - return ext.export_symbols + initfunc_name = "init" + string.split(ext.name,'.')[-1] + if initfunc_name not in ext.export_symbols: + ext.export_symbols.append(initfunc_name) + return ext.export_symbols def get_libraries (self, ext): """Return the list of libraries to link against when building a diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/cygwinccompiler.py d.p/distutils/cygwinccompiler.py --- distutils.orig/distutils/cygwinccompiler.py Thu Aug 3 10:59:58 2000 +++ d.p/distutils/cygwinccompiler.py Tue Aug 8 14:08:44 2000 @@ -44,16 +44,19 @@ __revision__ = "$Id: cygwinccompiler.py,v 1.4 2000/08/02 01:31:56 gward Exp $" -import os,sys +import os,sys,copy from distutils.unixccompiler import UnixCCompiler from distutils.file_util import write_file class CygwinCCompiler (UnixCCompiler): compiler_type = 'cygwin' - gcc_version = None - dllwrap_version = None - ld_version = None + obj_extension = ".o" + static_lib_extension = ".a" + shared_lib_extension = ".dll" + static_lib_format = "lib%s%s" + shared_lib_format = "%s%s" + exe_extension = ".exe" def __init__ (self, verbose=0, @@ -62,14 +65,16 @@ UnixCCompiler.__init__ (self, verbose, dry_run, force) - if check_config_h()<=0: + check_result = check_config_h() + self.debug_print("Python's GCC status: %s" % check_result) + if check_result[:2] <> "OK": self.warn( "Python's config.h doesn't seem to support your compiler. " "Compiling may fail because of undefined preprocessor macros.") (self.gcc_version, self.ld_version, self.dllwrap_version) = \ get_versions() - sys.stderr.write(self.compiler_type + ": gcc %s, ld %s, dllwrap %s\n" % + self.debug_print(self.compiler_type + ": gcc %s, ld %s, dllwrap %s\n" % (self.gcc_version, self.ld_version, self.dllwrap_version) ) @@ -117,9 +122,9 @@ extra_postargs=None, build_temp=None): - # use separate copies, so can modify the lists - extra_preargs = list(extra_preargs or []) - libraries = list(libraries or []) + # use separate copies, so we can modify the lists + extra_preargs = copy.copy(extra_preargs or []) + libraries = copy.copy(libraries or []) # Additional libraries libraries.extend(self.dll_libraries) @@ -241,11 +246,11 @@ compiling probably doesn't work. """ # return values - # 2: OK, python was compiled with GCC - # 1: OK, python's config.h mentions __GCC__ - # 0: uncertain, because we couldn't check it - # -1: probably not OK, because we didn't found it in config.h - # You could check check_config_h()>0 => OK + # "OK, python was compiled with GCC" + # "OK, python's config.h mentions __GCC__" + # "uncertain, because we couldn't check it" + # "not OK, because we didn't found __GCC__ in config.h" + # You could check check_config_h()[:2] == "OK" from distutils import sysconfig import string,sys @@ -254,7 +259,7 @@ if -1 == string.find(sys.version,"GCC"): pass # go to the next test else: - return 2 + return "OK, python was compiled with GCC" try: # It would probably better to read single lines to search. @@ -265,14 +270,14 @@ # is somewhere a #ifdef __GNUC__ or something similar if -1 == string.find(s,"__GNUC__"): - return -1 + return "not OK, because we didn't found __GCC__ in config.h" else: - return 1 + return "OK, python's config.h mentions __GCC__" except IOError: # if we can't read this file, we cannot say it is wrong # the compiler will complain later about this file as missing pass - return 0 + return "uncertain, because we couldn't check it" def get_versions(): """ Try to find out the versions of gcc, ld and dllwrap. diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/extension.py d.p/distutils/extension.py --- distutils.orig/distutils/extension.py Thu Aug 3 10:59:58 2000 +++ d.p/distutils/extension.py Tue Aug 8 14:26:26 2000 @@ -105,6 +105,6 @@ self.extra_objects = extra_objects or [] self.extra_compile_args = extra_compile_args or [] self.extra_link_args = extra_link_args or [] - self.export_symbols = export_symbols + self.export_symbols = export_symbols or [] # class Extension
On 08 August 2000, Rene Liebscher said:
Patch decription: [...]
OK, I've checked in all these changes.
- cygwin_compiler.py: [...] * changed result type of check_config_h() from int to string
I finally figured out how 'check_config_h()' should work; I've implemented it, tested it, and will check it in shortly. Here's the current code; please sanity-check it for me! (It works for me under Linux, at least.) ------------------------------------------------------------------------ def check_config_h(): """Check if the current Python installation (specifically, config.h) appears amenable to building extensions with GCC. Returns a tuple (status, details), where 'status' is one of the following constants: CONFIG_H_OK all is well, go ahead and compile CONFIG_H_NOTOK doesn't look good CONFIG_H_UNCERTAIN not sure -- unable to read config.h 'details' is a human-readable string explaining the situation. Note there are two ways to conclude "OK": either 'sys.version' contains the string "GCC" (implying that this Python was built with GCC), or the installed "config.h" contains the string "__GNUC__". """ # XXX since this function also checks sys.version, it's not strictly a # "config.h" check -- should probably be renamed... from distutils import sysconfig import string,sys # if sys.version contains GCC then python was compiled with # GCC, and the config.h file should be OK if string.find(sys.version,"GCC") >= 0: return (CONFIG_H_OK, "sys.version mentions 'GCC'") fn = sysconfig.get_config_h_filename() try: # It would probably better to read single lines to search. # But we do this only once, and it is fast enough f = open(fn) s = f.read() f.close() except IOError, exc: # if we can't read this file, we cannot say it is wrong # the compiler will complain later about this file as missing return (CONFIG_H_UNCERTAIN, "couldn't read '%s': %s" % (fn, exc.strerror)) else: # "config.h" contains an "#ifdef __GNUC__" or something similar if string.find(s,"__GNUC__") >= 0: return (CONFIG_H_OK, "'%s' mentions '__GNUC__'" % fn) else: return (CONFIG_H_NOTOK, "'%s' does not mention '__GNUC__'" % fn) ------------------------------------------------------------------------ The new use of 'check_config_h()' can be seen in the current CVS version of cygwinccompiler.py. Please let me know ASAP if this works for you; release 0.9.1 is long overdue, and we need to stop screwing around and get it out the door! Thanks -- Greg -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ Disclaimer: All rights reserved. Void where prohibited. Limit 1 per customer.
On 08 August 2000, Rene Liebscher said:
It is not as easy as you may think, I just spotted a potential problem before it could arise. If I would write ...append(",,%s," % filename) and filename would contain a blank ("/a b/file") then spawn would quote it as whole ",,/a b/file,". And this is probably wrong. So we should insert file names as single words in the list.
OK, you win. I think this situation sucks because of 1) MS-DOS' poorly-thought-out command-line model, which (as near as I can tell) has continued under Windows, and 2) Borland's abuse of Microsoft's fuzzy thinking. So your code makes the best of a bad situation. Oh well.
(Don't say blanks in file names are not allowed in python module names, THIS is a generic compiler class which could also be used for other purposes.)
Oh good, finally someone other than me is preaching this sermon -- thanks! ;-) Greg -- Greg Ward - nerd gward@python.net http://starship.python.net/~gward/ Never put off till tomorrow what you can put off till the day after tomorrow.
On 29 June 2000, Rene Liebscher said:
---- CygwinCCompiler ------------
OK, it should now be better to read. I also included your other changes (set_executables, build_temp.)
Looking better. Still some problems, which I will address below.
----- msvc hack -----------
I also had a look at this msvc hack in build_ext. I think it is now possible to get rid of it. It does several things which could be done in an other way or shouldn't be done at all.
First, it tries to find a def-file in the source directory if none is given. I think one should specify this explicitely, who knows if it is really the right file and moreover it overwrites any export_symbols given to Extension().
I have never been fond of this bit of code. I have always considered it unnecessary, given that 1) there is always exactly one symbol to be exported from a Python extension, and it is trivially derived from the extension name, and 2) we can generate an "/export:" option for that exported file. Somehow Thomas Heller (I think I can blame him ;-) talked me into leaving it in. But Thomas is on vacation now. (Hee hee hee.) I think I'll take a page from Guido's book and remove what I consider to be a broken feature. If nobody howls, it stays out. *poof!* it's gone.
Then it uses the module_name to create an export directive for the procedure 'init{modulename}'. It is possible to extract this name from the output filename (see (bcpp|cygwin)compiler.py ) I think this export parameter should be only used if neither a def file nor a list of symbols is given. (Then you could use your compiler classes also for non python dll's by specifying [] as export_symbols list.)
No: that would involve putting knowledge about Python extensions into a CCompiler class, which I strive to avoid. (Went to a lot of trouble this weekend to get knowledge of the sysconfig module out of UnixCCompiler, and I don't want to have to go through that again!) It is acceptable, but definitely to be avoided, to add platform- and compiler-specific hacks to build_ext.py. But I want to keep Python-extension-building hacks out of the CCompiler classes, because they can and should be useful for more than just building Python extensions. This, of course, is the remaining problem (that I see) with cygwinccompiler.py: it figures out symbols-to-export by looking at the output filename, which is flagrantly wrong (ie. totally specific to building Python extensions). *Please* fix this!
When I looked at the code of build_ext I found you are not using export_symbols as parameter to link_shared_object. Also you are not using any of this export_* values from the Extension class. This should be changed. But there is one point which is not clear. There is no parameter at link_shared_object for export_symbols_file. Either we add this or we change the semantics of the existing export_symbols a bit.
Yeah, those features were added so we could reduce (if not completely eliminate) the MSVC-specific hackery in build_ext.py. Looks like I never finished the job; kinda hard for me to do since I can't test it. I'd love to see a patch that does this, thought!
And finally it uses the build_temp path for it implib directive. We have now this new extra parameter to link_shared_library, so this is also not a problem.
Hey, good point. Can you submit a patch for that too? Greg -- Greg Ward - programmer-at-big gward@python.net http://starship.python.net/~gward/ It has just been discovered that research causes cancer in rats.
participants (3)
-
Greg Ward -
Norman Vine -
Rene Liebscher