FW: [issue2513] 64bit cross compilation on windows
FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited!
Thanks,
Mark
-----Original Message-----
From: Mark Hammond [mailto:report@bugs.python.org]
Sent: Sunday, 30 March 2008 6:01 PM
To: mhammond@users.sourceforge.net
Subject: [issue2513] 64bit cross compilation on windows
New submission from Mark Hammond
I wrote:
FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited!
While I have some people's attention I'd like to re-raise another issue I foresee for x64 builds. I've mentioned this over the last couple of months, but haven't got much of a response, so this seems like a reasonable opportunity to try one more time :) Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories. The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken. My proposal would help avoid breakage for existing build processes or tools that try to natively target x64, and give a reasonable story for build processes that explicitly support cross-compilation. I believe the costs are fairly small - disk space is cheap and the extra complexity in the VS project files should be reasonable. What do people think? I think it's time to take the approach of "silent ascent", so unless I hear objections I'll upload a new patch which implements this and also takes it into account for Distutils builds. Cheers, Mark
Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories.
Can you explain what you mean by "native platform"? I usually perform cross-builds for amd64, so would then this directory remain empty?
The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken.
The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries. If this is all about *just* Mozilla, then I'd be much more in favour than if it was for some theoretical package. So if Mozilla builds with that kind of setup, and works correctly - go for it. If you have to fix Mozilla anyway, consider fixing it so it looks in the amd64 directory. Regards, Martin
Martin quoting me:
Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories.
Can you explain what you mean by "native platform"? I usually perform cross-builds for amd64, so would then this directory remain empty?
By "native platform", I mean it contains binaries that are native for the current platform. IOW, on x86 platforms, PCBuild would contain x86 binaries, and binaries compiled for AMD64 would live in the AMD64 directory. If in a given directory tree, you *only* did a cross-compile, then yes, PCBuild would remain empty - but I don't think that is a problem; tools that know how to cross-compile (such as the distutils after my patch is applied) will not look in the PCBuild directory, and tools that do not cross-compile will be unable to use the x64 binaries from the x86 platform anyway. In other words, I can't think of anything that would break by having an empty PCBuild directory in a tree that only contains binaries that can't be executed on the current platform.
The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken.
The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries.
I agree. However, it is my assertion that there are very few build tools which expect the layout you describe, simply as the scheme has only been around for a few months, and only in Python 2.6 builds. Further, I assert that there are a greater number of build tools which do not support cross-compilation, but will build natively on x64 and expect 'PCBuild' to have libraries they can link with to create an x64 binary. I accept my assertions may be incorrect - in which case keeping the existing layout is indeed the best options. But I assert that my assertions are correct :)
If this is all about *just* Mozilla, then I'd be much more in favour than if it was for some theoretical package. So if Mozilla builds with that kind of setup, and works correctly - go for it. If you have to fix Mozilla anyway, consider fixing it so it looks in the amd64 directory.
Nope - it's not about *just* Mozilla at all, and I don't expect that I personally will be involved in any 64bit issues for that platform. That just happens to be one build process that I know about, but I except it does some "typical" things other tools might do - like executing the python executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to locate libraries to link with seems fairly likely to me. Even looking for the executable itself directly in PCBuild seems likely. On an x64 platform, such tools may well find the 32bit version, and it may even seem to work - but it is unlikely to be doing what they expect. Or they may find no binary at all. It seems that you previously agreed with this. From http://mail.python.org/pipermail/distutils-sig/2007-May/007668.html; you are replying to (I think) Kristján: | > I am baffled about why the build environment's layout matters, | > but once an .msi install can place the binaries in any | > old place it wants. The build structure doesn't have to | > reflect the final installed structure at all. | No. But still, people like to be able to "run" Python out of | a source check-out. This has been supported for a long time, | and more and more stuff was added to support it. For examples | within Python itself, see the support in distutils, getpathp.c, | PCbuild/rt.bat, and Tools/buildbot/*.bat. Reportedly (by | Mark), building Mozilla (the web browser) also "knows" | about PCbuild. Your comments exactly reflect my concern, and I believe them relevant for people working with native binaries on x64. But as implied above, I actually have zero personal interest in this at the moment - unlike the distutils cross-compile patch, which does scratch an itch of mine. It just seems it would make life easier for people down the track - but I'm happy to let it go if your position has changed or I have mis-understood it. Thanks, Mark
The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries.
I agree. However, it is my assertion that there are very few build tools which expect the layout you describe, simply as the scheme has only been around for a few months, and only in Python 2.6 builds.
That PCbuild exists and contains x86 binaries? This scheme has been in place ever since the PCbuild directory got created, several years ago!
Nope - it's not about *just* Mozilla at all, and I don't expect that I personally will be involved in any 64bit issues for that platform. That just happens to be one build process that I know about, but I except it does some "typical" things other tools might do - like executing the python executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to locate libraries to link with seems fairly likely to me. Even looking for the executable itself directly in PCBuild seems likely. On an x64 platform, such tools may well find the 32bit version, and it may even seem to work - but it is unlikely to be doing what they expect.
Unless the *want* the x86 binaries, of course, which always had been in this place.
Your comments exactly reflect my concern, and I believe them relevant for people working with native binaries on x64.
No, I never cared about what binaries are "native". Instead, I wanted to see those binaries in PCbuild which had been built last. If you just built the x86 binaries, PCbuild should contain x86 binaries, even if you were running Win64. This is different from what you propose, but only slightly so (but perhaps importantly, depending on the scenario).
But as implied above, I actually have zero personal interest in this at the moment - unlike the distutils cross-compile patch, which does scratch an itch of mine. It just seems it would make life easier for people down the track - but I'm happy to let it go if your position has changed or I have mis-understood it.
OK, if we don't have an actual use case, I suggest to leave things as is. People running into this problem should propose solutions themselves. Regards, Martin
Further, I assert that there are a greater number of build tools which do not support cross-compilation, but will build natively on x64 and expect 'PCBuild' to have libraries they can link with to create an x64 binary.
I'm with Martin on this one as well I think. If I understand correctly, you're proposing: PCbuild - may have contents of x86 or x64 depending on the build machine's architecture PCbuild/amd64 - always x64 PCbuild/x86 - always x86 And what we've currently got is: PCbuild/ - always x86 PCbuild/amd64 - always x64 I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation. I believe this is a Good Thing. If you want a 32-bit build, use the 'Win32' target platform in VS; if you want a 64-bit build, use 'x64'. Trent.
I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation.
Actually, I think it is slightly. IIUC, the AMD64 build currently assumes it can execute x86 executables in various places. To fix this, the build process for each of the platforms would be slightly different. My proposal would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to specified in build processes, with the knowledge it will work, regardless of what platform it is being run on. For example, with my idea, you could write code such as: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) popen(exe, ...) and expect it to work on all platforms. If I understand correctly, I now need to write something like: if "AMD64" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "amd64", "python.exe")) else if "32 bit" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) else: # hrm - I obviously dont know about new platforms yet... popen(exe, ...) In other words, I think there is real advantage to simple scripts knowing that './python' or './PCBuild/python.exe' will always be executable (and always give the "native" version if the platform does emulation) But I accept I'm waving appendages in the wind, and without a concrete use-case I will take Martin's advice and let it go. Cheers, Mark
Actually, I think it is slightly. IIUC, the AMD64 build currently assumes it can execute x86 executables in various places. To fix this, the build process for each of the platforms would be slightly different.
Why does that need fixing? The AMD64 build *can* execute x86 binaries, whether it is performed on Win32 or Win64. A problem only arises if the AMD64 build assumes that it can execute AMD64 binaries, and it is a cross-compilation. As a consequence, we must never execute the python.exe that we just built during the build process.
My proposal would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to specified in build processes, with the knowledge it will work, regardless of what platform it is being run on.
No to python.exe, yes to make_versioninfo.exe. make_versioninfo.exe is *always* an x86 binary, so it is always safe to execute on Windows. If you tried to execute PCBuild\python.exe, it would break my builds: I don't *have* x86 binaries in PCBuild when I cross-compile for AMD64. Instead, I've changed the build process to run HOST_PYTHON if that is set, and only fall back to PCBuild\python.exe if it isn't. Regards, Martin
participants (3)
-
"Martin v. Löwis"
-
Mark Hammond
-
Trent Nelson