[Distutils] Compiling / Installing extensions

[Greg Ward]

On 07 February 2000, Thomas Heller said:
> > In article <049a01bf716a$863ab650$4500a8c0@thomasnotebook>, Thomas
> > Heller <thomas.heller@ion-tof.com> writes
> > ...
> > >> 
> > >Only one change neccessary, in sysconfig.py line 150:
> > >    g['SO'] = '.dll'
> > >make this:
> > >    g['SO'] = '.pyd'
> > >
> > 
> > It's bad to do that here. This is supposed to be for general stuff.
> > DLLs are called .dll; python extensions are called .pyd.
> > The output extension is supposedly set in msvccompiler, but then
> > build_extension doesn't use it.
> Yes, you are right of course.

Ooh, this is getting hairy.  As it happens, I disagree with both of you
-- or rather, I agree with Thomas' original position, which was to hack
sysconfig.py.  Now I have to convince Thomas back around to his original 
view.  ;-)

My reasoning: sysconfig.py is Python-specific (it exists solely to parse
Python's config.h and Makefile), and therefore it's perfectly OK to put
code in there that only applies when building Python extensions.
Furthermore, the '_init_nt()' function is Windows-specific code in a
Python-specific module: what *better* place to do the hacks needed to
build Python extensions on Windows!  If only all the
{platform,language}-dependent stuff could be elegantly shoved in there.

Any other opinions?  Fred?  You wrote sysconfig.py, was setting g['SO']
to '.dll' right or wrong?  Should it be '.pyd' instead?

> I know this stuff (or at least I can read it in my msdn).
> The problem is:
> The linker generates an .exp and .lib file for every python extension
> built. Should msvccompiler.py delete these, or should install_ext
> delete them? Or should install_ext only copy the .pyd files
> instead of the whole tree?

It should be the job of MSVCCompiler to know what the compiler and
linker it drives do, including what temporary files they leave behind,
how to convince them to leave as few temporary files as possible, and
how to cleanup any that can't be suppressed.

The whole point of the "build" tree is to make installation a trivial
recursive copy; I really don't want to keep track of what files wind up
in "build" that are supposed to be installed, because by definition
they're *all* supposed to be installed.  The fact that compiler turds
wind up in the "build" tree is a bug in msvccompiler.py. (I could always
blame it on Microsoft, but I expect they document it somewhere.
Bastards.)

Here's an idea I've been toying with: give the "build" tree two tasks,
first to serve as an testing/installation staging area (its current
task) and second as an area for temporary build files.  Eg. after you
build a module distribution with extensions, the "build" tree would look
like

   -- build -- lib ------ .py files
               platlib -- .so files
               temp ----- .o, .a, and other temporary build by-products

or, to allow using the same source tree to build multiple architectures:

   -- build -- lib ---------------- .py files
               platlib.linux-x86 -- .so files
               platlib.winnt-x86 -- .pyd files
               temp.linux-x86 ----- .o, .a, etc.
               temp.winnt-x86 ----- .obj, .lib, .exp, etc.

This makes installation and testing only slightly less trivial, and now
cleanup also becomes a snap: "rm -rf build/temp.*" (coded in Python and
made available as a 'clean' command, of course).

Key question for the Window guys: is it possible to make MSVC++ put .exp 
files in a separate directory from the .pyd files produced at the same
time?  If not, that doesn't invalidate the above scheme, it just makes
it slightly less useful -- MSVCCompiler will still have to delete the
.exp (and other compiler/linker turds) after creating a .pyd.

> Is it true that /Zi to the compiler and /pdb:None to the linker
> does the same as /Z7 to the compiler? I'm not sure...
> 
> > Note that according to the standard we need to do stuff like using the
> > _d versions of dll/pyd names for our link.
> At least for python15.lib and pyton15_d.lib there is some
> magic in config.h, so this is automatic. Works also for /D_DEBUG.

Ugh.  OK, here's a deal: I'll put in the infrastructure for supporting a
"--debug" flag if you guys will figure out how to make it work with
MSVC++.  Fair enough?

> We should drop the line
>     self.add_library ( "python" + sys.version[0] + sys.version[2] )
> in msvccompiler.py.

Absoposidefinutely yes!  I guess by my evolving standard, that should go 
in the MSVC-specific code in build_ext.py.  (Am I going to live to
regret that decision?  sigh...)

        Greg
-- 
Greg Ward - Linux weenie                                gward@python.net
http://starship.python.net/~gward/
Think big -- pollute Lake Superior.