[Python-Dev] Adventures with x64, VS7 and VS8 on Windows

[Mark Hammond]

Hi Martin,

Way back on Monday, 21 May 2007, you wrote:
 
> This is an issue to be discussed for Python 2.6. I'm personally
> hesitant to have the "official" build infrastructure deviate 
> from the layout that has been in-use for so many years, as a lot 
> of things depend on it.
> 
> I don't find the need to have separate object directories convincing:
> For building the Win32/Win64 binaries, I have separate checkouts
> *anyway*, since all the add-on libraries would have to support
> multi-arch builds, but I think they don't.
...
> > * Re the x64 directory used by the PCBuild8 process.  IMO, it makes
> > sense to generate x64 binaries to their own directory - my expectation 
> > is that cross-compiling between platforms is a reasonable use-case, 
> > and we should support multiple achitectures for the same compiler 
> > version.
> 
> See above; I disagree.

[For additional context; the question was regarding where the output
binaries were created for each platform, and specifically if x64 build
should use something like 'PCBuild/x64']

I'm bringing this topic up again as I noticed the AMD64 builds for Python
2.6 create their output in the PCBuild/x64 directory, not directly into the
PCBuild directory.  When I raised this with Christian, he also expressed a
desire to be able to cross-compile in the same directory structure and was
unaware of this discussion (but I made him aware of it :)

You made excellent points about how tools currently recognize the PCBuild
directory, and we can't break them, and I agree.  I'd like to propose a
compromise that might be able to keep everyone happy.

The main problem I see with all platforms using 'PCBuild' is that in a
cross-compilation scenario, it is impossible for the distutils to determine
the location of the correct Python libraries to link with (stuff in its own
PYTHONHOME is no good; it's for the wrong platform).  Obviously, we can
change the distutils so that the location of the correct libraries can be
specified, but that implies all other build systems that currently assume
PCBuild must also change to work in a cross-compilation scenario.  While
those external tools *will* work today in a native build on x64, eventually
they may need to be upgraded to deal with cross-compilation - and this means
that all such tools will also be unable to automatically determine the
location of the libraries.  They will all need to take the library dir as a
user-specified option, which seems a pain (eg, buildbots would seem to need
site-specific configuration information to make this work).  If eventually
all such tools are likely to upgrade, it would be ideal if we can offer them
a way to upgrade so the correct libraries can be determined automatically,
as they can now for native builds.
 
My compromise proposal is this: Each platform builds in its own directory
(ie, PCBuild/x86_64, PCBuild/x86).  Every single build configuration has a
post-build step that copies the result of the build to the PCBuild
directory.  We then add an option to the distutils so that a cross-compile
platform can be specified and when it is, to use 'PCBuild/{platform}"
instead of PCBuild, and we encourage other tools to use the same directory
should they want to support "configure-less" cross-compilation (if distutils
on the trunk is not still trying to maintain b/w compat, it should just
*always* look in PCBuild/{platform}, and I'd suggest py3k not bother with
PCBuild at all; build systems will be touched to upgrade to py3k, so that
change isn't painful.  But I digress :)

The main advantage of the compromise is that it allows for the status quo to
remain in place - just continue to use separate source trees for each
platform, and only ever build a single platform in each tree, and tools are
free to have ways of specifying which directory should be used.  The
official build process need not change.  However, it also supports a way to
do cross-compilation in a way that build tools can *automatically* locate
the correct platform's libraries, and this will be particularly useful for
extension authors.

Would something like that be acceptable?

Thanks,

Mark