FW: [issue2513] 64bit cross compilation on windows
FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited!
Thanks,
Mark -----Original Message----- From: Mark Hammond [mailto:report@bugs.python.org] Sent: Sunday, 30 March 2008 6:01 PM To: mhammond@users.sourceforge.net Subject: [issue2513] 64bit cross compilation on windows
New submission from Mark Hammond mhammond@users.sourceforge.net:
I've taken the liberty of adding Trent, Christian and Martin to the nosy list as I know they are actively, if reluctantly interested in this.
This patch allows the distutils to cross-compile on Windows. It has been tested on x86 and amd64 platforms, with both platforms successfully able to natively and cross-compile extension modules and create binary distributions.
To cross-compile, specify '--plat-name' to the build command (valid values are 'win32', 'win-amd64' and 'win-ia64'). This option name was chosen to be consistent with the bdist_dumb command. I've included the docs I added below (which are also in the patch), but note that as with native compilation using distutils, it's not necessary to set any environment variables or do anything else special with your environment to make this work.
The patch also adds a x64 target for the 'bdist_wininst' target, which it creates as distutils/command/wininst-9.0-amd64.exe. This executable is necessary even for bdist_wininst to work natively on x64, but is still included here for simplicity.
To assist with testing, I've also added a distutils setup.py script to the PC/example_nt directory. This is capable of creating bdist_wininst executables for both native and cross platforms; 'setup.py build --platname=win-amd64 bdist_wininst' will create an amd64 installer on an x86 machine.
The patch has not been tested with a Visual Studio environment without cross-compile tools installed - it will obviously fail, but its not clear how ugly this failure will be.
Below is the text I added to docs/distutils/builtdist.rst:
Cross-compiling on Windows =====================
Starting with Python 2.6, distutils is capable of cross-compiling between Windows platforms. In practice, this means that with the correct tools installed, you can use a 32bit version of Windows to create 64bit extensions and vice-versa.
To build for an alternate platform, specify the :option:`--plat-name` option to the build command. Valid values are currently 'win32', 'win-amd64' and 'win-ia64'. For example, on a 32bit version of Windows, you could execute::
python setup.py build --plat-name=win-amd64
to build a 64bit version of your extension. The Windows Installers also support this option, so the command::
python setup.py build --plat-name=win-amd64 bdist_wininst
would create a 64bit installation executable on your 32bit version of Windows.
Note that by default, Visual Studio 2008 does not install 64bit compilers or tools. You may need to reexecute the Visual Studio setup process and select these tools.
---------- assignee: mhammond components: Distutils files: windows-cross-compile.patch keywords: 64bit, patch messages: 64743 nosy: Trent.Nelson, ctheune, loewis, mhammond severity: normal status: open title: 64bit cross compilation on windows type: feature request versions: Python 2.6 Added file: http://bugs.python.org/file9900/windows-cross-compile.patch
__________________________________ Tracker report@bugs.python.org http://bugs.python.org/issue2513 __________________________________
I wrote:
FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited!
While I have some people's attention I'd like to re-raise another issue I foresee for x64 builds. I've mentioned this over the last couple of months, but haven't got much of a response, so this seems like a reasonable opportunity to try one more time :)
Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories.
The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken. My proposal would help avoid breakage for existing build processes or tools that try to natively target x64, and give a reasonable story for build processes that explicitly support cross-compilation. I believe the costs are fairly small - disk space is cheap and the extra complexity in the VS project files should be reasonable.
What do people think? I think it's time to take the approach of "silent ascent", so unless I hear objections I'll upload a new patch which implements this and also takes it into account for Distutils builds.
Cheers,
Mark
Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories.
Can you explain what you mean by "native platform"? I usually perform cross-builds for amd64, so would then this directory remain empty?
The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken.
The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries.
If this is all about *just* Mozilla, then I'd be much more in favour than if it was for some theoretical package. So if Mozilla builds with that kind of setup, and works correctly - go for it. If you have to fix Mozilla anyway, consider fixing it so it looks in the amd64 directory.
Regards, Martin
Martin quoting me:
Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories.
Can you explain what you mean by "native platform"? I usually perform cross-builds for amd64, so would then this directory remain empty?
By "native platform", I mean it contains binaries that are native for the current platform. IOW, on x86 platforms, PCBuild would contain x86 binaries, and binaries compiled for AMD64 would live in the AMD64 directory.
If in a given directory tree, you *only* did a cross-compile, then yes, PCBuild would remain empty - but I don't think that is a problem; tools that know how to cross-compile (such as the distutils after my patch is applied) will not look in the PCBuild directory, and tools that do not cross-compile will be unable to use the x64 binaries from the x86 platform anyway. In other words, I can't think of anything that would break by having an empty PCBuild directory in a tree that only contains binaries that can't be executed on the current platform.
The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken.
The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries.
I agree. However, it is my assertion that there are very few build tools which expect the layout you describe, simply as the scheme has only been around for a few months, and only in Python 2.6 builds. Further, I assert that there are a greater number of build tools which do not support cross-compilation, but will build natively on x64 and expect 'PCBuild' to have libraries they can link with to create an x64 binary.
I accept my assertions may be incorrect - in which case keeping the existing layout is indeed the best options. But I assert that my assertions are correct :)
If this is all about *just* Mozilla, then I'd be much more in favour than if it was for some theoretical package. So if Mozilla builds with that kind of setup, and works correctly - go for it. If you have to fix Mozilla anyway, consider fixing it so it looks in the amd64 directory.
Nope - it's not about *just* Mozilla at all, and I don't expect that I personally will be involved in any 64bit issues for that platform. That just happens to be one build process that I know about, but I except it does some "typical" things other tools might do - like executing the python executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to locate libraries to link with seems fairly likely to me. Even looking for the executable itself directly in PCBuild seems likely. On an x64 platform, such tools may well find the 32bit version, and it may even seem to work - but it is unlikely to be doing what they expect. Or they may find no binary at all.
It seems that you previously agreed with this. From http://mail.python.org/pipermail/distutils-sig/2007-May/007668.html; you are replying to (I think) Kristján:
| > I am baffled about why the build environment's layout matters, | > but once an .msi install can place the binaries in any | > old place it wants. The build structure doesn't have to | > reflect the final installed structure at all.
| No. But still, people like to be able to "run" Python out of | a source check-out. This has been supported for a long time, | and more and more stuff was added to support it. For examples | within Python itself, see the support in distutils, getpathp.c, | PCbuild/rt.bat, and Tools/buildbot/*.bat. Reportedly (by | Mark), building Mozilla (the web browser) also "knows" | about PCbuild.
Your comments exactly reflect my concern, and I believe them relevant for people working with native binaries on x64. But as implied above, I actually have zero personal interest in this at the moment - unlike the distutils cross-compile patch, which does scratch an itch of mine. It just seems it would make life easier for people down the track - but I'm happy to let it go if your position has changed or I have mis-understood it.
Thanks,
Mark
The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries.
I agree. However, it is my assertion that there are very few build tools which expect the layout you describe, simply as the scheme has only been around for a few months, and only in Python 2.6 builds.
That PCbuild exists and contains x86 binaries? This scheme has been in place ever since the PCbuild directory got created, several years ago!
Nope - it's not about *just* Mozilla at all, and I don't expect that I personally will be involved in any 64bit issues for that platform. That just happens to be one build process that I know about, but I except it does some "typical" things other tools might do - like executing the python executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to locate libraries to link with seems fairly likely to me. Even looking for the executable itself directly in PCBuild seems likely. On an x64 platform, such tools may well find the 32bit version, and it may even seem to work - but it is unlikely to be doing what they expect.
Unless the *want* the x86 binaries, of course, which always had been in this place.
Your comments exactly reflect my concern, and I believe them relevant for people working with native binaries on x64.
No, I never cared about what binaries are "native". Instead, I wanted to see those binaries in PCbuild which had been built last. If you just built the x86 binaries, PCbuild should contain x86 binaries, even if you were running Win64.
This is different from what you propose, but only slightly so (but perhaps importantly, depending on the scenario).
But as implied above, I actually have zero personal interest in this at the moment - unlike the distutils cross-compile patch, which does scratch an itch of mine. It just seems it would make life easier for people down the track - but I'm happy to let it go if your position has changed or I have mis-understood it.
OK, if we don't have an actual use case, I suggest to leave things as is. People running into this problem should propose solutions themselves.
Regards, Martin
Further, I assert that there are a greater number of build tools which do not support cross-compilation, but will build natively on x64 and expect 'PCBuild' to have libraries they can link with to create an x64 binary.
I'm with Martin on this one as well I think. If I understand correctly, you're proposing:
PCbuild - may have contents of x86 or x64 depending on the build machine's architecture PCbuild/amd64 - always x64 PCbuild/x86 - always x86
And what we've currently got is:
PCbuild/ - always x86 PCbuild/amd64 - always x64
I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation. I believe this is a Good Thing. If you want a 32-bit build, use the 'Win32' target platform in VS; if you want a 64-bit build, use 'x64'.
Trent.
I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation.
Actually, I think it is slightly. IIUC, the AMD64 build currently assumes it can execute x86 executables in various places. To fix this, the build process for each of the platforms would be slightly different. My proposal would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to specified in build processes, with the knowledge it will work, regardless of what platform it is being run on.
For example, with my idea, you could write code such as:
exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) popen(exe, ...)
and expect it to work on all platforms. If I understand correctly, I now need to write something like:
if "AMD64" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "amd64", "python.exe")) else if "32 bit" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) else: # hrm - I obviously dont know about new platforms yet... popen(exe, ...)
In other words, I think there is real advantage to simple scripts knowing that './python' or './PCBuild/python.exe' will always be executable (and always give the "native" version if the platform does emulation)
But I accept I'm waving appendages in the wind, and without a concrete use-case I will take Martin's advice and let it go.
Cheers,
Mark
Actually, I think it is slightly. IIUC, the AMD64 build currently assumes it can execute x86 executables in various places. To fix this, the build process for each of the platforms would be slightly different.
Why does that need fixing? The AMD64 build *can* execute x86 binaries, whether it is performed on Win32 or Win64.
A problem only arises if the AMD64 build assumes that it can execute AMD64 binaries, and it is a cross-compilation. As a consequence, we must never execute the python.exe that we just built during the build process.
My proposal would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to specified in build processes, with the knowledge it will work, regardless of what platform it is being run on.
No to python.exe, yes to make_versioninfo.exe. make_versioninfo.exe is *always* an x86 binary, so it is always safe to execute on Windows. If you tried to execute PCBuild\python.exe, it would break my builds: I don't *have* x86 binaries in PCBuild when I cross-compile for AMD64.
Instead, I've changed the build process to run HOST_PYTHON if that is set, and only fall back to PCBuild\python.exe if it isn't.
Regards, Martin
participants (3)
-
"Martin v. Löwis" -
Mark Hammond -
Trent Nelson