[Distutils] [Python-Dev] Adventures with x64, VS7 and VS8 on Windows

["Martin v. Löwis"]

> * In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the
> PCBuild directory), the python.exe project fails with:
> 
>     pythoncore fatal error LNK1112: module machine type 'X86' conflicts with
> target machine type 'AMD64'
> 
>   is this a known issue, or am I doing something wrong?

You are likely doing something wrong:
a) I assume it's VS 7.1 (i.e. VS.NET 2003); VS 2002 is not supported
   at all
b) you probably didn't install vsextcomp, but you should.
   In fact, you don't need all of it, but you do need the cl.exe and
   link.exe wrappers it comes with - they dispatch to the proper
   tools from the SDK
c) in case it isn't clear: you also need an AMD64 compiler, e.g.
   from the platform SDK.
   Unfortunately, Microsoft keeps changing the registry settings for
   the SDK, so vsextcomp only knows about some selected SDKs. If
   that causes a problem, please let me know.

> * The PCBuild8 project files appear to work without modification (I only
> tried native compilation here though, not a cross-compile) - however, unlike
> the PCBuild directory, they place all binaries in a 'PCBuild8/x64'
> directory.  While this means that its possible to build for multiple
> architectures from the same source tree, it makes life harder for tools like
> 'distutils' - eg, distutils already knows about the 'PCBuild' directory, but
> it knows nothing about either PCBuild8 or PCBuild8/x64.

This is an issue to be discussed for Python 2.6. I'm personally hesitant
to have the "official" build infrastructure deviate from the layout that
has been in-use for so many years, as a lot of things depend on it.

I don't find the need to have separate object directories convincing:
For building the Win32/Win64 binaries, I have separate checkouts
*anyway*, since all the add-on libraries would have to support
multi-arch builds, but I think they don't.

> A number of other build processes also know to look inside a PCBuild
> directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we
> should merge PCBuild8 into PCBuild.

Right - PCbuild8 should not get formalized. It probably should continue
to be maintained.

For 2.6, the first question to answer is: what compiler should it use?

I would personally like to see Python "skip" VS 2005 altogether,
as it will be soon superceded by Orcas. Unfortunately, it's unclear
how long Microsoft will need to release Orcas (and also, when Python
2.6 will be released), so I would like to defer that question by
a few months.

> I understand that PCBuild8 is not "official", but in the
> assumption that future versions of Python will use a compiler later than
> VS7, it makes sense to me to clean this up now - what are others opinions on
> this?

Not "official" really only means "not used to build the official
binaries" - just like PC/VC6. It's still (somewhat) maintained.

As for cleaning it up - see above. I would *really* like to skip
VS 2005 altogether, as I expect that soon after we decide to use
VS 2005, Microsoft will replace it with the next release, stop
supporting VS 2005, take the free compiler off the next, and
so on (just like they did for VS 2003, soon after we decided to
use it for 2.5).

> * Re the x64 directory used by the PCBuild8 process.  IMO, it makes sense to
> generate x64 binaries to their own directory - my expectation is that
> cross-compiling between platforms is a reasonable use-case, and we should
> support multiple achitectures for the same compiler version. 

See above; I disagree. First, "multiple architectures" only means x86,
AMD64, and Itanium, and I would like to drop "official" Itanium binaries
from 2.6 (even though they could continue to be supported in the build
process). Then, even if the Python build itself support multiple
simultaneous architectures, the extension modules don't all (correct
me if I'm wrong).

> This would
> mean formalizing the x64 directory in both 'PCBuild' and distutils, and
> leaving other external build processes to update as they support x64 builds.
> Does this make sense?  Would this fatally break other scripts used for
> packaging (eg, the MSI framework)?

The MSI packaging would need to be changed, certainly. It currently
detects the architecture it needs to package by looking at the file
type of python.exe; that would have to be changed to give it an
explicit parameter what architecture to package, or have it package
all architectures it can find.

> * Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned
> short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler
> versions.  VS8 defines this as wchar_t, which I'm struggling to find a
> formal definition for beyond being 2 bytes.

In C or in C++? In C++, wchar_t is a builtin type, just like short, int,
long. So there is no further formal definition.

In C (including C99), wchar_t ought to be defined in stddef.h.

> My problem is that code which
> assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails.  I
> believe the intent on Windows has always been "PyUNICODE == 'native
> unicode'" - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a
> version specific definition of PyUNICODE as wchar_t)?

I'd rather make it a platform-specific definition (for platform=Windows
API). Correct me if I'm wrong, but isn't wchar_t also available in VS
2003 (and even in VC6?). And doesn't it have the "right" definition in
all these compilers?

So +1 for setting Py_UNICODE to wchar_t on Windows.

> * Finally, as something from left-field which may well take 12 months or
> more to pull off - but would there be any interest to moving the Windows
> build process to a cygwin environment based on the existing autoconf
> scripts?

What compiler would you use there? I very much like using the VS
debugger when developing on Windows, so that capability should not
go away.

Regards,
Martin