Compiling / Installing extensions
It seems, with Robins latest patch compiling extensions under windows goes smoothly. Some points are still nonstandard: 1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed? 2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong? 3. It would really be nice if there were a possibility to compile debug versions. Thomas Heller
In article <026901bf6f25$cd3d4140$4500a8c0@thomasnotebook>, Thomas Heller <thomas.heller@ion-tof.com> writes
It seems, with Robins latest patch compiling extensions under windows goes smoothly.
Some points are still nonstandard:
1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?
2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?
3. It would really be nice if there were a possibility to compile debug versions.
Thomas Heller
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://www.python.org/mailman/listinfo/distutils-sig
Python picks up .pyd's or .dll's for extensions. We could just change _shared_lib_ext to '.pyd'. -- Robin Becker
In article <03be01bf6f47$2ec3ac80$4500a8c0@thomasnotebook>, Thomas Heller <thomas.heller@ion-tof.com> writes
Python picks up .pyd's or .dll's for extensions. We could just change _shared_lib_ext to '.pyd'. But this does change nothing. Only after I changed g['SO'] = '.dll' to g['SO'] = '.pyd' in sysconfig.py: _init_nt()
Have to look after this. Thomas ....
mmmmhhhh this code is dusty -- Robin Becker
Thomas Heller writes:
1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?
The PYD extension is good because it indicates that it isn't used as a "normal" DLL; this is useful because it makes file management easier.
2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?
On Unix, linking doesn't typically leave a bunch of extra stuff. I don't even know what an EXP file is! ;)
3. It would really be nice if there were a possibility to compile debug versions.
Agreed, but I'm just a Unix weenie. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
Thomas Heller writes:
1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?
The PYD extension is good because it indicates that it isn't used as a "normal" DLL; this is useful because it makes file management easier. It also makes possible (I think) to wrap the python interface for thirdparty.dll into thirdparty.pyd. So it should be changed. Greg, what do you think?
2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?
On Unix, linking doesn't typically leave a bunch of extra stuff. On windows, it's usually not even possible to suppress this extra stuff. Depending on the linker/compiler flags, even more files are generated. It seems distutils should only copy the target files, not this extra stuff. Greg ?
I don't even know what an EXP file is! ;) Same for the typical windows programmer ;)
3. It would really be nice if there were a possibility to compile debug versions.
Agreed, but I'm just a Unix weenie. What do you mean? You don't need to debug under Unix? Is the debug version the default? Or what?
Thomas
I said:
Agreed, but I'm just a Unix weenie.
Thomas Heller writes:
What do you mean? You don't need to debug under Unix? Is the debug version the default? Or what?
Thomas, I meant that I write perfect code on the first pass, every time, of course. Unless it's in an email. ;) I was just saying that there should be some way for distutils to provide an option to do that. It should be there for all supported platforms, not just Windows. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
On 04 February 2000, Thomas Heller said:
It seems, with Robins latest patch compiling extensions under windows goes smoothly.
Great! I got a bit lost in the flurry of patches-to-patches, so could one of you mail me either a definitive patch or a replacement version of msvccompiler.py with the "let's go find the compiler" code included? (Send it to gward@python.net so I'll get it at home.)
1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?
Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, msvccompiler.py is *not* the place to fix this. The CCompiler classes are meant to apply to more than just compiling Python extensions, but rather to know "everything" about compiling and linking C on a given platform/compiler. Unfortunately, the rest of the Distutils tries really hard to be platform-neutral, which means this kind of stuff -- which is specific to building Python extensions on a given platform -- falls between the cracks. My preferred solution is to put platform-dependent code into build_ext.py, as you can see in the current kludge for handling .def files. That reminds me, I've been convinced on several occasions that .def files are not necessary (since Python extensions really only need to export one symbol, 'init_foo()' or whatever it is, which can easily be specified on the compiler command-line). I'd love to see that MSVC-specific kludge stripped out of build_ext.py, but I think this requires some other changes: * notion of "list of symbols to export" when building a shared object or library (in CCompiler and all subclasses -- it would just be ignored in UnixCCompiler) * build_ext passes ['init_%s' % extension_name] for that * MSVCCompiler generates the appropriate command-line options for cl.exe to export the listed symbols If any of you intrepid Windows hackers would like to tackle this, be my guest! And feel free to do whatever cleanup you like in msvccompiler.py; there is definitely some cruft in there.
2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?
Wow, I had no idea this was happening. That should be the job of MSVCCompiler to clean up, since it "knows everything" about the compiler and platform, eg. what crufty turds it drops behind after compiling or linking something. Silly me, I blithely assumed that no compiler would be so stupid as to leave unnecessary temporary files around after a build. I should never have underestimated Microsoft's capacity for mind-numbingly stupid code.
3. It would really be nice if there were a possibility to compile debug versions.
Shouldn't be too hard; just need to add: * debug flag to CCompiler instances * debug flag to CCompiler methods (the usual "method param overrides instance attribute" paradigm I've used everywhere in those classes) * UnixCCompiler sticks "-g" into the command line if debug is true * MSVCCompiler sticks "**something**" into the command line if debug is true * build_ext and build_lib should take 'debug' options, which can be set on the command line I think if all this is done, then setup.py build_ext --debug will do what you want on either Unix or Windows. If I don't get a patch for it this weekend, I'll do it myself... Greg
In article <20000204230137.B22123@cnri.reston.va.us>, Greg Ward <gward@cnri.reston.va.us> writes
On 04 February 2000, Thomas Heller said:
It seems, with Robins latest patch compiling extensions under windows goes smoothly.
Great! I got a bit lost in the flurry of patches-to-patches, so could one of you mail me either a definitive patch or a replacement version of msvccompiler.py with the "let's go find the compiler" code included? (Send it to gward@python.net so I'll get it at home.)
Thomas has the latest working version
1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?
Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, msvccompiler.py is *not* the place to fix this. The CCompiler classes are meant to apply to more than just compiling Python extensions, but rather to know "everything" about compiling and linking C on a given platform/compiler. Unfortunately, the rest of the Distutils tries really hard to be platform-neutral, which means this kind of stuff -- which is specific to building Python extensions on a given platform -- falls between the cracks. ....
agreed
My preferred solution is to put platform-dependent code into build_ext.py, as you can see in the current kludge for handling .def files.
That reminds me, I've been convinced on several occasions that .def files are not necessary (since Python extensions really only need to export one symbol, 'init_foo()' or whatever it is, which can easily be specified on the compiler command-line). I'd love to see that MSVC-specific kludge stripped out of build_ext.py, but I think this requires some other changes:
* notion of "list of symbols to export" when building a shared object or library (in CCompiler and all subclasses -- it would just be ignored in UnixCCompiler) * build_ext passes ['init_%s' % extension_name] for that * MSVCCompiler generates the appropriate command-line options for cl.exe to export the listed symbols
This is certainly doable. The compile.py from used to do that.
If any of you intrepid Windows hackers would like to tackle this, be my guest! And feel free to do whatever cleanup you like in msvccompiler.py; there is definitely some cruft in there.
2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?
Wow, I had no idea this was happening. That should be the job of MSVCCompiler to clean up, since it "knows everything" about the compiler and platform, eg. what crufty turds it drops behind after compiling or linking something. Silly me, I blithely assumed that no compiler would be so stupid as to leave unnecessary temporary files around after a build. I should never have underestimated Microsoft's capacity for mind-numbingly stupid code.
3. It would really be nice if there were a possibility to compile debug versions.
Shouldn't be too hard; just need to add:
* debug flag to CCompiler instances * debug flag to CCompiler methods (the usual "method param overrides instance attribute" paradigm I've used everywhere in those classes) * UnixCCompiler sticks "-g" into the command line if debug is true * MSVCCompiler sticks "**something**" into the command line if debug is true * build_ext and build_lib should take 'debug' options, which can be set on the command line
I think if all this is done, then
setup.py build_ext --debug
will do what you want on either Unix or Windows. If I don't get a patch for it this weekend, I'll do it myself...
Greg
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://www.python.org/mailman/listinfo/distutils-sig
-- Robin Becker
In article <9qtONDAat+m4Ewsg@jessikat.demon.co.uk>, Robin Becker <robin@jessikat.demon.co.uk> writes ... the change to get the initXXX exported currently needs to be done in build_extensions at the point where the check for a .def file is done. In my view it is wrong for the compiler specific stuff to be anywhere other than the specific compiler file, but there you go. so in build_ext.py line 225 should look like if self.compiler.compiler_type == 'msvc': extra_args.append('/export:init%s'%extension_name) What's the point of all the definitions in msvccompiler if they aren't used? -- Robin Becker
On 07 February 2000, Robin Becker said:
In my view it is wrong for the compiler specific stuff to be anywhere other than the specific compiler file, but there you go.
It's a tradeoff between kludging the Python-extension-specific, platform-neutral code or kludging the Python-extension-ignorant, platform-specific code. I prefer the former, because Python-extension-specific code has fewer applications outside of the Distutils. The application-neutral, platform-specific code (MSVCCompiler and UnixCCompiler) could be more broadly applicable, so I'm trying to keep it "pure".
so in build_ext.py line 225 should look like
if self.compiler.compiler_type == 'msvc': extra_args.append('/export:init%s'%extension_name)
OK, except you mean line 238 in the current CVS version. ;-) At any rate, done. Any Windows people who feel a deep attachment for .def files better speak up now. More seriously, is there any reason a Python extension module needs to export any symbols other than "init%s" % extension_name ???
What's the point of all the definitions in msvccompiler if they aren't used?
Hey, I said it was crufty code. But now that I look at it, I only see one class attribute ('_exe_ext') that's not used. That's not too bothersome. So what *are* you referring to? Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ Cheops' Law: Nothing *ever* gets built on schedule or within budget.
so in build_ext.py line 225 should look like
if self.compiler.compiler_type == 'msvc': extra_args.append('/export:init%s'%extension_name)
OK, except you mean line 238 in the current CVS version. ;-) At any rate, done. Any Windows people who feel a deep attachment for .def files better speak up now. More seriously, is there any reason a Python extension module needs to export any symbols other than "init%s" % extension_name ??? I suggest: if self.compiler.compiler_type == 'msvc': def_file = build_info.get ('def_file') if def_file is None: source_dir = os.path.dirname (sources[0]) ext_base = (string.split (extension_name, '.'))[-1] def_file = os.path.join (source_dir, "%s.def" % ext_base) if not os.path.exists (def_file): self.warn ("file '%s' not found: " % def_file + "might have problems building DLL") def_file = None
if def_file is not None: extra_args.append ('/DEF:' + def_file) else: modname = string.split (extension_name, '.')[-1] extra_args.append('/export:init%s'%modname) split(extension_name) is needed for extensions living in package directories. If the developer bothers to write a def file, it should be used. Otherwise guess the exported symbol. Note: For normal python extensions, only this single export is used. This may be different for COM-modules for example... Thomas
On 08 February 2000, Thomas Heller said:
I suggest: if self.compiler.compiler_type == 'msvc': def_file = build_info.get ('def_file') if def_file is None: source_dir = os.path.dirname (sources[0]) ext_base = (string.split (extension_name, '.'))[-1] def_file = os.path.join (source_dir, "%s.def" % ext_base) if not os.path.exists (def_file): self.warn ("file '%s' not found: " % def_file + "might have problems building DLL") def_file = None
if def_file is not None: extra_args.append ('/DEF:' + def_file) else: modname = string.split (extension_name, '.')[-1] extra_args.append('/export:init%s'%modname)
split(extension_name) is needed for extensions living in package directories.
D'oh! Thanks, fixed it (not in CVS yet though).
If the developer bothers to write a def file, it should be used. Otherwise guess the exported symbol.
Note: For normal python extensions, only this single export is used. This may be different for COM-modules for example...
Oh bother. OK, if it's really needed. ;-( A question for the crowd (especially Windows extension developers) before I give in: have you ever needed to define a .def file to link your extensions? Or, more bluntly, is Thomas correct, that a COM-aware extension might need to export more symbols than 'initfoo'? No offense Thomas, I just want to make sure that this ugly hack really is needed. Greg -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ It has just been discovered that research causes cancer in rats.
In article <20000208202001.A487@beelzebub>, Greg Ward <gward@python.net> writes ...
A question for the crowd (especially Windows extension developers) before I give in: have you ever needed to define a .def file to link your extensions? Or, more bluntly, is Thomas correct, that a COM-aware extension might need to export more symbols than 'initfoo'? No offense Thomas, I just want to make sure that this ugly hack really is needed.
Greg
It may be convenient to just provide a list of the exported symbols. The _numpy extension has quite a few things it wants to export. A disadavantage is that adding a new exportable requires editing two files. The .def may get forgotten. It won't matter till it's acutally needed. In the _numpy case it may be sometime before the C api actually gets used. -- Robin Becker
1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?
Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, msvccompiler.py is *not* the place to fix this. The CCompiler classes are meant to apply to more than just compiling Python extensions, but rather to know "everything" about compiling and linking C on a given platform/compiler. Unfortunately, the rest of the Distutils tries really hard to be platform-neutral, which means this kind of stuff -- which is specific to building Python extensions on a given platform -- falls between the cracks.
Only one change neccessary, in sysconfig.py line 150: g['SO'] = '.dll' make this: g['SO'] = '.pyd'
2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?
Wow, I had no idea this was happening. That should be the job of MSVCCompiler to clean up, since it "knows everything" about the compiler and platform, eg. what crufty turds it drops behind after compiling or linking something. Silly me, I blithely assumed that no compiler would be so stupid as to leave unnecessary temporary files around after a build. I should never have underestimated Microsoft's capacity for mind-numbingly stupid code.
3. It would really be nice if there were a possibility to compile debug versions.
Shouldn't be too hard; just need to add:
* debug flag to CCompiler instances * debug flag to CCompiler methods (the usual "method param overrides instance attribute" paradigm I've used everywhere in those classes) * UnixCCompiler sticks "-g" into the command line if debug is true * MSVCCompiler sticks "**something**" into the command line if debug is true * build_ext and build_lib should take 'debug' options, which can be set on the command line
I think if all this is done, then
setup.py build_ext --debug
will do what you want on either Unix or Windows. If I don't get a patch for it this weekend, I'll do it myself...
Greg If we start to compile debug version, we will have to struggle with the .pdb file, the 'program database' containing debugging information. What do other windows developers think? Should we link with /pdb:None, which will write the debugging info into the executable? Or should we
The current msvccompiler.py I sent to you adds the /INCREMENTAL:No flag to the linker, which will suppress the .ilk file (used for incremental linking). The generation of .exp and .lib files cannot be suppressed by command line flags. If you want to use msvccompiler.py in a way you described above, we cannot simply delete the .exp and .lib files because they may be needed for dependent modules to be build. Wouldn't it be better to change install_ext to only copy the expected files? live with the .pdb file in the same directory as the .pyd? Thomas
In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas Heller <thomas.heller@ion-tof.com> writes ...
Only one change neccessary, in sysconfig.py line 150: g['SO'] = '.dll' make this: g['SO'] = '.pyd'
It's bad to do that here. This is supposed to be for general stuff. DLLs are called .dll; python extensions are called .pyd. The output extension is supposedly set in msvccompiler, but then build_extension doesn't use it.
The current msvccompiler.py I sent to you adds the /INCREMENTAL:No flag to the linker, which will suppress the .ilk file (used for incremental linking).
The generation of .exp and .lib files cannot be suppressed by command line flags.
If you want to use msvccompiler.py in a way you described above, we cannot simply delete the .exp and .lib files because they may be needed for dependent modules to be build. Wouldn't it be better to change install_ext to only copy the expected files?
....
In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas Heller <thomas.heller@ion-tof.com> writes ...
Only one change neccessary, in sysconfig.py line 150: g['SO'] = '.dll' make this: g['SO'] = '.pyd'
It's bad to do that here. This is supposed to be for general stuff. DLLs are called .dll; python extensions are called .pyd. The output extension is supposedly set in msvccompiler, but then build_extension doesn't use it. Yes, you are right of course.
The current msvccompiler.py I sent to you adds the /INCREMENTAL:No flag to the linker, which will suppress the .ilk file (used for incremental linking).
The generation of .exp and .lib files cannot be suppressed by command line flags.
If you want to use msvccompiler.py in a way you described above, we cannot simply delete the .exp and .lib files because they may be needed for dependent modules to be build. Wouldn't it be better to change install_ext to only copy the expected files?
....
From the MSDN docs [deleted]
I know this stuff (or at least I can read it in my msdn). The problem is: The linker generates an .exp and .lib file for every python extension built. Should msvccompiler.py delete these, or should install_ext delete them? Or should install_ext only copy the .pyd files instead of the whole tree?
We could try and use /Z7 type debugging instead of /Zi
/Z7 C 7.0- Compatible Produces an .OBJ file and an .EXE file containing line numbers and full symbolic debugging information for use with the debugger. The symbolic debugging information includes the names and types of variables, as well as functions and line numbers.
Is it true that /Zi to the compiler and /pdb:None to the linker does the same as /Z7 to the compiler? I'm not sure...
Note that according to the standard we need to do stuff like using the _d versions of dll/pyd names for our link. At least for python15.lib and pyton15_d.lib there is some magic in config.h, so this is automatic. Works also for /D_DEBUG. We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.
Thomas
In article <053e01bf7180$bdee8020$4500a8c0@thomasnotebook>, Thomas Heller <thomas.heller@ion-tof.com> writes ...
Yes, you are right of course. ...
being right doesn't help. The proper thing to do is somehow to get the right value for .pyds and or .dlls Greg does the higher level give any indication whether this is a python only thing or a normal dll?
....
From the MSDN docs [deleted]
I know this stuff (or at least I can read it in my msdn). The problem is: The linker generates an .exp and .lib file for every python extension built. Should msvccompiler.py delete these, or should install_ext delete them? Or should install_ext only copy the .pyd files instead of the whole tree?
As I understand it the .exp files are only needed when we want to do complicated (2 stage) linking a dll imports from and exports to another.
We could try and use /Z7 type debugging instead of /Zi
/Z7 C 7.0- Compatible Produces an .OBJ file and an .EXE file containing line numbers and full symbolic debugging information for use with the debugger. The symbolic debugging information includes the names and types of variables, as well as functions and line numbers.
Is it true that /Zi to the compiler and /pdb:None to the linker does the same as /Z7 to the compiler? I'm not sure...
Note that according to the standard we need to do stuff like using the _d versions of dll/pyd names for our link. At least for python15.lib and pyton15_d.lib there is some magic in config.h, so this is automatic. Works also for /D_DEBUG. We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.
Thomas ...
You do cl -c -Z7 dingo.c .... link dingo.obj ...other.obj /incremental:no /pdb:NONE /debug /debugtype:cv /export:initdingo /out:dingo.pyd and no pdb file. I checked and it seems as though this results in files that can still be debugged provided you can find the .c files. the python_d.lib needs to be used in the link phase. -- Robin Becker
On 07 February 2000, Robin Becker said:
being right doesn't help. The proper thing to do is somehow to get the right value for .pyds and or .dlls
Greg does the higher level give any indication whether this is a python only thing or a normal dll?
For now, the Distutils command classes only build DLLs (shared libraries or shared objects, whatever you want to call them) that are Python extensions. The compiler classes don't know that, though, and there's no guarantee that it'll stay that way in the future. As I've said, I favour hacking sysconfig.py. That means the PYD/DLL distinction is entirely filename based. Will that work?
You do cl -c -Z7 dingo.c .... link dingo.obj ...other.obj /incremental:no /pdb:NONE /debug /debugtype:cv /export:initdingo /out:dingo.pyd
and no pdb file. I checked and it seems as though this results in files that can still be debugged provided you can find the .c files.
Oh, I forgot to mention: I strongly favour putting debug info in the executable or DLL or .so or whatever it's called. It's been that way in Unix since time immemorial, so IMHO it's good enough for Python extensions under Windows. ;-) Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ This quote intentionally left blank.
The Numeric package itself has _numpy which exports lots of things. I don't seriously object to changing sysconfig.py. Go ahead. Since the .pyd only gets done in sysconfig it follows that _shared_lib_ext = '.pyd' in msvccompiler doesn't get used so then it follows that shared_library_filename isn't being used properly. I say go for the -Z7 debugging mode. I would point out that this is essentially a legacy mode for M$ as it represents C7.0 (the CodeView era) and as with all legacy things it will be withdrawn as and when Bill sees fit. I can only find one output location option /out: and it seems that as soon as you specify one export that .lib and .exp files get created alongside the output file. So the new logic should be that the compiler shoves its .obj files into temp and the linker intially does likewise. A final stage copies wanted files .pyd .lib into the platlib.winnt.x86 or whatever. can we have a keep list which defaults to .pyd, but can be added to by an option. Then _numpy could add .lib As a final note; if you want to have a single temp area, we need different object extensions eg .obj and .dobj for optimised and debug versions. -- Robin Becker
[Robin]
The Numeric package itself has _numpy which exports lots of things.
I don't seriously object to changing sysconfig.py. Go ahead. Since the .pyd only gets done in sysconfig it follows that _shared_lib_ext = '.pyd' in msvccompiler doesn't get used so then it follows that shared_library_filename isn't being used properly.
I say go for the -Z7 debugging mode. I would point out that this is essentially a legacy mode for M$ as it represents C7.0 (the CodeView era) and as with all legacy things it will be withdrawn as and when Bill sees fit. We could hope this -Z7 would stay for a while. Even now you can generate mapfiles (wasn't this the symdeb era?).
I can only find one output location option /out: and it seems that as soon as you specify one export that .lib and .exp files get created alongside the output file. So the new logic should be that the compiler shoves its .obj files into temp and the linker intially does likewise. A final stage copies wanted files .pyd .lib into the platlib.winnt.x86 or whatever. See my previous post about the /implib: linker flag.
can we have a keep list which defaults to .pyd, but can be added to by an option. Then _numpy could add .lib
As a final note; if you want to have a single temp area, we need different object extensions eg .obj and .dobj for optimised and debug versions.
The usual way (also used for core python) is to have separate directories: temp-debug and temp-release. And to have separate pyd filenames: module.pyd and module_d.pyd.
In article <020101bf7228$f94d8a50$4500a8c0@thomasnotebook>, Thomas Heller <thomas.heller@ion-tof.com> writes
[Robin]
The Numeric package itself has _numpy which exports lots of things.
I don't seriously object to changing sysconfig.py. Go ahead. Since the .pyd only gets done in sysconfig it follows that _shared_lib_ext = '.pyd' in msvccompiler doesn't get used so then it follows that shared_library_filename isn't being used properly.
I say go for the -Z7 debugging mode. I would point out that this is essentially a legacy mode for M$ as it represents C7.0 (the CodeView era) and as with all legacy things it will be withdrawn as and when Bill sees fit. We could hope this -Z7 would stay for a while. Even now you can generate mapfiles (wasn't this the symdeb era?).
I can only find one output location option /out: and it seems that as soon as you specify one export that .lib and .exp files get created alongside the output file. So the new logic should be that the compiler shoves its .obj files into temp and the linker intially does likewise. A final stage copies wanted files .pyd .lib into the platlib.winnt.x86 or whatever. See my previous post about the /implib: linker flag.
can we have a keep list which defaults to .pyd, but can be added to by an option. Then _numpy could add .lib
As a final note; if you want to have a single temp area, we need different object extensions eg .obj and .dobj for optimised and debug versions.
The usual way (also used for core python) is to have separate directories: temp-debug and temp-release. And to have separate pyd filenames: module.pyd and module_d.pyd.
.... agreed! Will you do a version after pulling the CVS code? -- Robin Becker
On 07 February 2000, Thomas Heller said:
In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas Heller <thomas.heller@ion-tof.com> writes ...
Only one change neccessary, in sysconfig.py line 150: g['SO'] = '.dll' make this: g['SO'] = '.pyd'
It's bad to do that here. This is supposed to be for general stuff. DLLs are called .dll; python extensions are called .pyd. The output extension is supposedly set in msvccompiler, but then build_extension doesn't use it. Yes, you are right of course.
Ooh, this is getting hairy. As it happens, I disagree with both of you -- or rather, I agree with Thomas' original position, which was to hack sysconfig.py. Now I have to convince Thomas back around to his original view. ;-) My reasoning: sysconfig.py is Python-specific (it exists solely to parse Python's config.h and Makefile), and therefore it's perfectly OK to put code in there that only applies when building Python extensions. Furthermore, the '_init_nt()' function is Windows-specific code in a Python-specific module: what *better* place to do the hacks needed to build Python extensions on Windows! If only all the {platform,language}-dependent stuff could be elegantly shoved in there. Any other opinions? Fred? You wrote sysconfig.py, was setting g['SO'] to '.dll' right or wrong? Should it be '.pyd' instead?
I know this stuff (or at least I can read it in my msdn). The problem is: The linker generates an .exp and .lib file for every python extension built. Should msvccompiler.py delete these, or should install_ext delete them? Or should install_ext only copy the .pyd files instead of the whole tree?
It should be the job of MSVCCompiler to know what the compiler and linker it drives do, including what temporary files they leave behind, how to convince them to leave as few temporary files as possible, and how to cleanup any that can't be suppressed. The whole point of the "build" tree is to make installation a trivial recursive copy; I really don't want to keep track of what files wind up in "build" that are supposed to be installed, because by definition they're *all* supposed to be installed. The fact that compiler turds wind up in the "build" tree is a bug in msvccompiler.py. (I could always blame it on Microsoft, but I expect they document it somewhere. Bastards.) Here's an idea I've been toying with: give the "build" tree two tasks, first to serve as an testing/installation staging area (its current task) and second as an area for temporary build files. Eg. after you build a module distribution with extensions, the "build" tree would look like -- build -- lib ------ .py files platlib -- .so files temp ----- .o, .a, and other temporary build by-products or, to allow using the same source tree to build multiple architectures: -- build -- lib ---------------- .py files platlib.linux-x86 -- .so files platlib.winnt-x86 -- .pyd files temp.linux-x86 ----- .o, .a, etc. temp.winnt-x86 ----- .obj, .lib, .exp, etc. This makes installation and testing only slightly less trivial, and now cleanup also becomes a snap: "rm -rf build/temp.*" (coded in Python and made available as a 'clean' command, of course). Key question for the Window guys: is it possible to make MSVC++ put .exp files in a separate directory from the .pyd files produced at the same time? If not, that doesn't invalidate the above scheme, it just makes it slightly less useful -- MSVCCompiler will still have to delete the .exp (and other compiler/linker turds) after creating a .pyd.
Is it true that /Zi to the compiler and /pdb:None to the linker does the same as /Z7 to the compiler? I'm not sure...
Note that according to the standard we need to do stuff like using the _d versions of dll/pyd names for our link. At least for python15.lib and pyton15_d.lib there is some magic in config.h, so this is automatic. Works also for /D_DEBUG.
Ugh. OK, here's a deal: I'll put in the infrastructure for supporting a "--debug" flag if you guys will figure out how to make it work with MSVC++. Fair enough?
We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.
Absoposidefinutely yes! I guess by my evolving standard, that should go in the MSVC-specific code in build_ext.py. (Am I going to live to regret that decision? sigh...) Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ Think big -- pollute Lake Superior.
Key question for the Window guys: is it possible to make MSVC++ put .exp files in a separate directory from the .pyd files produced at the same time? If not, that doesn't invalidate the above scheme, it just makes it slightly less useful -- MSVCCompiler will still have to delete the .exp (and other compiler/linker turds) after creating a .pyd. Yes, the /implib:build\temp\mymodule.lib puts the .lib AND the .exp files in a certain directory. So build_ext (or whatever) can prune the temp-directory.
Ugh. OK, here's a deal: I'll put in the infrastructure for supporting a "--debug" flag if you guys will figure out how to make it work with MSVC++. Fair enough?
Fair enough. I'm waiting for your patch.
We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.
Absoposidefinutely yes! I guess by my evolving standard, that should go in the MSVC-specific code in build_ext.py.
It is NOT NEEDED at all! The correct pythonxxx.lib will automatically be included on windows. Thomas
On 08 February 2000, Thomas Heller said:
Yes, the /implib:build\temp\mymodule.lib puts the .lib AND the .exp files in a certain directory. So build_ext (or whatever) can prune the temp-directory.
OK, noted. Let's wait to put it in until after I've changed the "build" commands to build to that temporary directory. Oh, I think you said something about dealing with name conflicts between debugging and non-debugging versions. For temp files, I think the obvious solution is to make "debugging" another "architecture", e.g. build ---- temp.linux-x86 temp.linux-x86-debug temp.winnt-x86 temp.winnt-x86-debug etc. It's not so clear what to do for the final product (.pyd or .so files)... but now that I think of it, if I'm going to allow multi-architecture builds in the same place (someone tell me this is a useful feature please!), then I should carry through and make the "build/platlib" directory name the platform, eg. "build/platlib.linux-x86". Keep on thinking of "debugging mode" as an aspect of the "architecture", and problem solved. Seem reasonable?
Ugh. OK, here's a deal: I'll put in the infrastructure for supporting a "--debug" flag if you guys will figure out how to make it work with MSVC++. Fair enough? Fair enough. I'm waiting for your patch.
Done. It's in CVS now. (I take it from earlier comments that this is enough for you, and a code snapshot is not necessary.)
We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.
Absoposidefinutely yes! I guess by my evolving standard, that should go in the MSVC-specific code in build_ext.py. It is NOT NEEDED at all! The correct pythonxxx.lib will automatically be included on windows.
Cool! That's good news. I hate to keep sounding disbelieving of you, but if there is a "25 words or less" explanation of why that works, would you mind filling me in? (Mainly I'm curious -- if you tell me not to get worried about it, it's complicated but it just works, then I'll probably trust you. But I *do* like to know what's going on behind the scenes...) Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ A man wrapped up in himself makes a very small package.
We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.
Absoposidefinutely yes! I guess by my evolving standard, that should go in the MSVC-specific code in build_ext.py. It is NOT NEEDED at all! The correct pythonxxx.lib will automatically be included on windows.
Cool! That's good news. I hate to keep sounding disbelieving of you, but if there is a "25 words or less" explanation of why that works, would you mind filling me in? (Mainly I'm curious -- if you tell me not to get worried about it, it's complicated but it just works, then I'll probably trust you. But I *do* like to know what's going on behind the scenes...)
MSVC has a pragma which allow to specify in the C-source file libraries to link with. Looking at config.h (from 1.5.2): /* So nobody needs to specify the .lib in their Makefile any more */ #ifdef _DEBUG #pragma comment(lib,"python15_d.lib") #else #pragma comment(lib,"python15.lib") Quoting from MSDN: #pragma comment( comment-type [, commentstring] ) Places a comment record into an object file or executable file. The comment-type is one of five predefined identifiers, described below, that specify the type of comment record. The optional commentstring is a string literal that provides additional information for some comment types. Because commentstring is a string literal, it obeys all the rules for string literals with respect to escape characters, embedded quotation marks ("), and concatenation. [...] lib Places a library-search record in the object file. This comment type must be accompanied by a commentstring parameter containing the name (and possibly the path) of the library that you want the linker to search. Since the library name precedes the default library-search records in the object file, the linker searches for this library just as if you had named it on the command line. You can place multiple library-search records in the same source file; each record appears in the object file in the same order in which it is encountered in the source file. [End of quote]
participants (5)
-
Fred L. Drake, Jr. -
Greg Ward -
Greg Ward -
Robin Becker -
Thomas Heller