In article <03be01bf6f47$2ec3ac80$4500a8c0@thomasnotebook>, Thomas Heller writes

...

Python picks up .pyd's or .dll's for extensions. We could just change _shared_lib_ext to '.pyd'. But this does change nothing. Only after I changed g['SO'] = '.dll' to g['SO'] = '.pyd' in sysconfig.py: _init_nt()

Have to look after this. Thomas ....

mmmmhhhh this code is dusty -- Robin Becker

Thomas Heller writes:

...

1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?

The PYD extension is good because it indicates that it isn't used as a "normal" DLL; this is useful because it makes file management easier. It also makes possible (I think) to wrap the python interface for thirdparty.dll into thirdparty.pyd. So it should be changed. Greg, what do you think?

...

2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?

On Unix, linking doesn't typically leave a bunch of extra stuff. On windows, it's usually not even possible to suppress this extra stuff. Depending on the linker/compiler flags, even more files are generated. It seems distutils should only copy the target files, not this extra stuff. Greg ?

I don't even know what an EXP file is! ;) Same for the typical windows programmer ;)

...

3. It would really be nice if there were a possibility to compile debug versions.

Agreed, but I'm just a Unix weenie. What do you mean? You don't need to debug under Unix? Is the debug version the default? Or what?

Thomas

I was just saying that there should be some way for distutils to provide an option to do that. It should be there for all supported platforms, not just Windows. Yes.

Thomas

In article <20000204230137.B22123@cnri.reston.va.us>, Greg Ward writes

On 04 February 2000, Thomas Heller said:

...

It seems, with Robins latest patch compiling extensions under windows goes smoothly.

Great! I got a bit lost in the flurry of patches-to-patches, so could one of you mail me either a definitive patch or a replacement version of msvccompiler.py with the "let's go find the compiler" code included? (Send it to gward@python.net so I'll get it at home.)

Thomas has the latest working version

...

1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?

Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, msvccompiler.py is *not* the place to fix this. The CCompiler classes are meant to apply to more than just compiling Python extensions, but rather to know "everything" about compiling and linking C on a given platform/compiler. Unfortunately, the rest of the Distutils tries really hard to be platform-neutral, which means this kind of stuff -- which is specific to building Python extensions on a given platform -- falls between the cracks. ....

agreed

My preferred solution is to put platform-dependent code into build_ext.py, as you can see in the current kludge for handling .def files.

That reminds me, I've been convinced on several occasions that .def files are not necessary (since Python extensions really only need to export one symbol, 'init_foo()' or whatever it is, which can easily be specified on the compiler command-line). I'd love to see that MSVC-specific kludge stripped out of build_ext.py, but I think this requires some other changes:

* notion of "list of symbols to export" when building a shared object or library (in CCompiler and all subclasses -- it would just be ignored in UnixCCompiler) * build_ext passes ['init_%s' % extension_name] for that * MSVCCompiler generates the appropriate command-line options for cl.exe to export the listed symbols

This is certainly doable. The compile.py from used to do that.

If any of you intrepid Windows hackers would like to tackle this, be my guest! And feel free to do whatever cleanup you like in msvccompiler.py; there is definitely some cruft in there.

...

2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?

Wow, I had no idea this was happening. That should be the job of MSVCCompiler to clean up, since it "knows everything" about the compiler and platform, eg. what crufty turds it drops behind after compiling or linking something. Silly me, I blithely assumed that no compiler would be so stupid as to leave unnecessary temporary files around after a build. I should never have underestimated Microsoft's capacity for mind-numbingly stupid code.

...

3. It would really be nice if there were a possibility to compile debug versions.

Shouldn't be too hard; just need to add:

* debug flag to CCompiler instances * debug flag to CCompiler methods (the usual "method param overrides instance attribute" paradigm I've used everywhere in those classes) * UnixCCompiler sticks "-g" into the command line if debug is true * MSVCCompiler sticks "**something**" into the command line if debug is true * build_ext and build_lib should take 'debug' options, which can be set on the command line

I think if all this is done, then

setup.py build_ext --debug

will do what you want on either Unix or Windows. If I don't get a patch for it this weekend, I'll do it myself...

Greg

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://www.python.org/mailman/listinfo/distutils-sig

-- Robin Becker

On 07 February 2000, Robin Becker said:

In my view it is wrong for the compiler specific stuff to be anywhere other than the specific compiler file, but there you go.

It's a tradeoff between kludging the Python-extension-specific, platform-neutral code or kludging the Python-extension-ignorant, platform-specific code. I prefer the former, because Python-extension-specific code has fewer applications outside of the Distutils. The application-neutral, platform-specific code (MSVCCompiler and UnixCCompiler) could be more broadly applicable, so I'm trying to keep it "pure".

so in build_ext.py line 225 should look like

if self.compiler.compiler_type == 'msvc': extra_args.append('/export:init%s'%extension_name)

OK, except you mean line 238 in the current CVS version. ;-) At any rate, done. Any Windows people who feel a deep attachment for .def files better speak up now. More seriously, is there any reason a Python extension module needs to export any symbols other than "init%s" % extension_name ???

What's the point of all the definitions in msvccompiler if they aren't used?

Hey, I said it was crufty code. But now that I look at it, I only see one class attribute ('_exe_ext') that's not used. That's not too bothersome. So what *are* you referring to? Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ Cheops' Law: Nothing *ever* gets built on schedule or within budget.

On 08 February 2000, Thomas Heller said:

I suggest: if self.compiler.compiler_type == 'msvc': def_file = build_info.get ('def_file') if def_file is None: source_dir = os.path.dirname (sources[0]) ext_base = (string.split (extension_name, '.'))[-1] def_file = os.path.join (source_dir, "%s.def" % ext_base) if not os.path.exists (def_file): self.warn ("file '%s' not found: " % def_file + "might have problems building DLL") def_file = None

if def_file is not None: extra_args.append ('/DEF:' + def_file) else: modname = string.split (extension_name, '.')[-1] extra_args.append('/export:init%s'%modname)

split(extension_name) is needed for extensions living in package directories.

D'oh! Thanks, fixed it (not in CVS yet though).

If the developer bothers to write a def file, it should be used. Otherwise guess the exported symbol.

Note: For normal python extensions, only this single export is used. This may be different for COM-modules for example...

Oh bother. OK, if it's really needed. ;-( A question for the crowd (especially Windows extension developers) before I give in: have you ever needed to define a .def file to link your extensions? Or, more bluntly, is Thomas correct, that a COM-aware extension might need to export more symbols than 'initfoo'? No offense Thomas, I just want to make sure that this ugly hack really is needed. Greg -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ It has just been discovered that research causes cancer in rats.

...

1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed?

Definitely -- the "foo.pyd wraps foo.dll" argument is compelling. FWIW, msvccompiler.py is *not* the place to fix this. The CCompiler classes are meant to apply to more than just compiling Python extensions, but rather to know "everything" about compiling and linking C on a given platform/compiler. Unfortunately, the rest of the Distutils tries really hard to be platform-neutral, which means this kind of stuff -- which is specific to building Python extensions on a given platform -- falls between the cracks.

Only one change neccessary, in sysconfig.py line 150: g['SO'] = '.dll' make this: g['SO'] = '.pyd'

...

2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong?

Wow, I had no idea this was happening. That should be the job of MSVCCompiler to clean up, since it "knows everything" about the compiler and platform, eg. what crufty turds it drops behind after compiling or linking something. Silly me, I blithely assumed that no compiler would be so stupid as to leave unnecessary temporary files around after a build. I should never have underestimated Microsoft's capacity for mind-numbingly stupid code.

...

3. It would really be nice if there were a possibility to compile debug versions.

Shouldn't be too hard; just need to add:

* debug flag to CCompiler instances * debug flag to CCompiler methods (the usual "method param overrides instance attribute" paradigm I've used everywhere in those classes) * UnixCCompiler sticks "-g" into the command line if debug is true * MSVCCompiler sticks "**something**" into the command line if debug is true * build_ext and build_lib should take 'debug' options, which can be set on the command line

I think if all this is done, then

setup.py build_ext --debug

will do what you want on either Unix or Windows. If I don't get a patch for it this weekend, I'll do it myself...

Greg If we start to compile debug version, we will have to struggle with the .pdb file, the 'program database' containing debugging information. What do other windows developers think? Should we link with /pdb:None, which will write the debugging info into the executable? Or should we

The current msvccompiler.py I sent to you adds the /INCREMENTAL:No flag to the linker, which will suppress the .ilk file (used for incremental linking). The generation of .exp and .lib files cannot be suppressed by command line flags. If you want to use msvccompiler.py in a way you described above, we cannot simply delete the .exp and .lib files because they may be needed for dependent modules to be build. Wouldn't it be better to change install_ext to only copy the expected files? live with the .pdb file in the same directory as the .pyd? Thomas

In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas Heller writes ...

...

...

Only one change neccessary, in sysconfig.py line 150: g['SO'] = '.dll' make this: g['SO'] = '.pyd'

It's bad to do that here. This is supposed to be for general stuff. DLLs are called .dll; python extensions are called .pyd. The output extension is supposedly set in msvccompiler, but then build_extension doesn't use it. Yes, you are right of course.

...

...

The current msvccompiler.py I sent to you adds the /INCREMENTAL:No flag to the linker, which will suppress the .ilk file (used for incremental linking).

The generation of .exp and .lib files cannot be suppressed by command line flags.

If you want to use msvccompiler.py in a way you described above, we cannot simply delete the .exp and .lib files because they may be needed for dependent modules to be build. Wouldn't it be better to change install_ext to only copy the expected files?

....

...

From the MSDN docs [deleted]

I know this stuff (or at least I can read it in my msdn). The problem is: The linker generates an .exp and .lib file for every python extension built. Should msvccompiler.py delete these, or should install_ext delete them? Or should install_ext only copy the .pyd files instead of the whole tree?

We could try and use /Z7 type debugging instead of /Zi

/Z7 C 7.0- Compatible Produces an .OBJ file and an .EXE file containing line numbers and full symbolic debugging information for use with the debugger. The symbolic debugging information includes the names and types of variables, as well as functions and line numbers.

Is it true that /Zi to the compiler and /pdb:None to the linker does the same as /Z7 to the compiler? I'm not sure...

Note that according to the standard we need to do stuff like using the _d versions of dll/pyd names for our link. At least for python15.lib and pyton15_d.lib there is some magic in config.h, so this is automatic. Works also for /D_DEBUG. We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.

Thomas

On 07 February 2000, Robin Becker said:

being right doesn't help. The proper thing to do is somehow to get the right value for .pyds and or .dlls

Greg does the higher level give any indication whether this is a python only thing or a normal dll?

For now, the Distutils command classes only build DLLs (shared libraries or shared objects, whatever you want to call them) that are Python extensions. The compiler classes don't know that, though, and there's no guarantee that it'll stay that way in the future. As I've said, I favour hacking sysconfig.py. That means the PYD/DLL distinction is entirely filename based. Will that work?

You do cl -c -Z7 dingo.c .... link dingo.obj ...other.obj /incremental:no /pdb:NONE /debug /debugtype:cv /export:initdingo /out:dingo.pyd

and no pdb file. I checked and it seems as though this results in files that can still be debugged provided you can find the .c files.

Oh, I forgot to mention: I strongly favour putting debug info in the executable or DLL or .so or whatever it's called. It's been that way in Unix since time immemorial, so IMHO it's good enough for Python extensions under Windows. ;-) Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ This quote intentionally left blank.

[Robin]

The Numeric package itself has _numpy which exports lots of things.

I don't seriously object to changing sysconfig.py. Go ahead. Since the .pyd only gets done in sysconfig it follows that _shared_lib_ext = '.pyd' in msvccompiler doesn't get used so then it follows that shared_library_filename isn't being used properly.

I say go for the -Z7 debugging mode. I would point out that this is essentially a legacy mode for M$ as it represents C7.0 (the CodeView era) and as with all legacy things it will be withdrawn as and when Bill sees fit. We could hope this -Z7 would stay for a while. Even now you can generate mapfiles (wasn't this the symdeb era?).

I can only find one output location option /out: and it seems that as soon as you specify one export that .lib and .exp files get created alongside the output file. So the new logic should be that the compiler shoves its .obj files into temp and the linker intially does likewise. A final stage copies wanted files .pyd .lib into the platlib.winnt.x86 or whatever. See my previous post about the /implib: linker flag.

can we have a keep list which defaults to .pyd, but can be added to by an option. Then _numpy could add .lib

As a final note; if you want to have a single temp area, we need different object extensions eg .obj and .dobj for optimised and debug versions.

The usual way (also used for core python) is to have separate directories: temp-debug and temp-release. And to have separate pyd filenames: module.pyd and module_d.pyd.

.... agreed! Will you do a version after pulling the CVS code? Yes. I'm only waiting for greg's code with the debug flag.

Thomas

Key question for the Window guys: is it possible to make MSVC++ put .exp files in a separate directory from the .pyd files produced at the same time? If not, that doesn't invalidate the above scheme, it just makes it slightly less useful -- MSVCCompiler will still have to delete the .exp (and other compiler/linker turds) after creating a .pyd. Yes, the /implib:build\temp\mymodule.lib puts the .lib AND the .exp files in a certain directory. So build_ext (or whatever) can prune the temp-directory.

Ugh. OK, here's a deal: I'll put in the infrastructure for supporting a "--debug" flag if you guys will figure out how to make it work with MSVC++. Fair enough?

Fair enough. I'm waiting for your patch.

...

We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.

Absoposidefinutely yes! I guess by my evolving standard, that should go in the MSVC-specific code in build_ext.py.

It is NOT NEEDED at all! The correct pythonxxx.lib will automatically be included on windows. Thomas

...

...

...

We should drop the line self.add_library ( "python" + sys.version[0] + sys.version[2] ) in msvccompiler.py.

Absoposidefinutely yes! I guess by my evolving standard, that should go in the MSVC-specific code in build_ext.py. It is NOT NEEDED at all! The correct pythonxxx.lib will automatically be included on windows.

Cool! That's good news. I hate to keep sounding disbelieving of you, but if there is a "25 words or less" explanation of why that works, would you mind filling me in? (Mainly I'm curious -- if you tell me not to get worried about it, it's complicated but it just works, then I'll probably trust you. But I *do* like to know what's going on behind the scenes...)

MSVC has a pragma which allow to specify in the C-source file libraries to link with. Looking at config.h (from 1.5.2): /* So nobody needs to specify the .lib in their Makefile any more */ #ifdef _DEBUG #pragma comment(lib,"python15_d.lib") #else #pragma comment(lib,"python15.lib") Quoting from MSDN: #pragma comment( comment-type [, commentstring] ) Places a comment record into an object file or executable file. The comment-type is one of five predefined identifiers, described below, that specify the type of comment record. The optional commentstring is a string literal that provides additional information for some comment types. Because commentstring is a string literal, it obeys all the rules for string literals with respect to escape characters, embedded quotation marks ("), and concatenation. [...] lib Places a library-search record in the object file. This comment type must be accompanied by a commentstring parameter containing the name (and possibly the path) of the library that you want the linker to search. Since the library name precedes the default library-search records in the object file, the linker searches for this library just as if you had named it on the command line. You can place multiple library-search records in the same source file; each record appears in the object file in the same order in which it is encountered in the source file. [End of quote]