Bug in build system for cross-platform builds
Hi all,
I’ve been working on developing Python builds for mobile platforms, and I’m looking for some help resolving a bug in Python’s build system.
The problem affects cross-platform builds - builds where you are compiling python for a CPU architecture other than the one on the machine that is doing the compilation. This requirement stems from supporting mobile platforms (iOS, Android etc) where you compile on your laptop, then ship the compiled binary to the device.
In the Python 3.5 dev cycle, Issue 22359 [1] was addressed, fixing parallel builds. However, as a side effect, this patch broke (as far as I can tell) *all* cross platform builds. This was reported in issue 22625 [2].
Since that time, the problem has gotten slightly worse; the addition of changeset 95566 [3] and 95854 [4] has cemented the problem. I’ve been able to hack together a fix that enables me to get a set of binaries, but the patch is essentially reverting 22359, and making some (very dubious) assumptions about the order in which things are built.
Autoconf et al aren’t my strong suit; I was hoping someone might be able to help me resolve this issue.
Yours, Russ Magee %-)
[1] http://bugs.python.org/issue22359 [2] http://bugs.python.org/issue22625 [3] https://hg.python.org/cpython/rev/565b96093ec8 [4] https://hg.python.org/cpython/rev/02e3bf65b2f8
On 15 February 2016 at 08:24, Russell Keith-Magee russell@keith-magee.com wrote:
Hi all,
I’ve been working on developing Python builds for mobile platforms, and I’m looking for some help resolving a bug in Python’s build system.
The problem affects cross-platform builds - builds where you are compiling python for a CPU architecture other than the one on the machine that is doing the compilation. This requirement stems from supporting mobile platforms (iOS, Android etc) where you compile on your laptop, then ship the compiled binary to the device.
In the Python 3.5 dev cycle, Issue 22359 [1] was addressed, fixing parallel builds. However, as a side effect, this patch broke (as far as I can tell) *all* cross platform builds. This was reported in issue 22625 [2].
Since that time, the problem has gotten slightly worse; the addition of changeset 95566 [3] and 95854 [4] has cemented the problem. I’ve been able to hack together a fix that enables me to get a set of binaries, but the patch is essentially reverting 22359, and making some (very dubious) assumptions about the order in which things are built.
Autoconf et al aren’t my strong suit; I was hoping someone might be able to help me resolve this issue.
Would you mind answering my question in https://bugs.python.org/issue22625#msg247652? In particular, how did cross-compiling previously work before these changes. AFAIK Python builds a preliminary Python executable which is executed on the host to complete the final build. So how do you differentiate between host and target compilers etc?
On Tue, Feb 16, 2016 at 5:22 AM, Martin Panter vadmium+py@gmail.com wrote:
On 15 February 2016 at 08:24, Russell Keith-Magee russell@keith-magee.com wrote:
Hi all,
I’ve been working on developing Python builds for mobile platforms, and
I’m
looking for some help resolving a bug in Python’s build system.
The problem affects cross-platform builds - builds where you are
compiling
python for a CPU architecture other than the one on the machine that is doing the compilation. This requirement stems from supporting mobile platforms (iOS, Android etc) where you compile on your laptop, then ship
the
compiled binary to the device.
In the Python 3.5 dev cycle, Issue 22359 [1] was addressed, fixing
parallel
builds. However, as a side effect, this patch broke (as far as I can
tell)
*all* cross platform builds. This was reported in issue 22625 [2].
Since that time, the problem has gotten slightly worse; the addition of changeset 95566 [3] and 95854 [4] has cemented the problem. I’ve been
able
to hack together a fix that enables me to get a set of binaries, but the patch is essentially reverting 22359, and making some (very dubious) assumptions about the order in which things are built.
Autoconf et al aren’t my strong suit; I was hoping someone might be able
to
help me resolve this issue.
Would you mind answering my question in https://bugs.python.org/issue22625#msg247652? In particular, how did cross-compiling previously work before these changes. AFAIK Python builds a preliminary Python executable which is executed on the host to complete the final build. So how do you differentiate between host and target compilers etc?
In order to build for a host platform, you have to compile for a local platform first - for example, to compile an iOS ARM64 binary, you have to compile for OS X x86_64 first. This gives you a local platform version of Python you can use when building the iOS version.
Early in the Makefile, the variable PYTHON_FOR_BUILD is set. This points at the CPU-local version of Python that can be invoked, which is used for module builds, and for compiling the standard library source code. This is set by —host and —build flags to configure, plus the use of CC and LDFLAGS environment variables to point at the compiler and libraries for the platform you’re compiling for, and a PATH variable that provides the local platform’s version of Python.
There are two places where special handling is required: the compilation and execution of the parser generator, and _freeze_importlib. In both cases, the tool needs to be compiled for the local platform, and then executed. Historically (i.e., Py3.4 and earlier), this has been done by spawning a child MAKE to compile the tool; this runs the compilation phase with the local CPU environment, before returning to the master makefile and executing the tool. By spawning the child MAKE, you get a “clean” environment, so the tool is built natively. However, as I understand it, it causes problems with parallel builds due to race conditions on build rules. The change in Python3.5 simplified the rule so that child MAKE calls weren’t used, but that means that pgen and _freeze_importlib are compiled for ARM64, so they won’t run on the local platform.
As best as I can work out, the solution is to:
(1) Include the parser generator and _freeze_importlib as part of the artefacts of local platform. That way, you could use the version of pgen and _freeze_importlib that was compiled as part of the local platform build. At present, pgen and _freeze_importlib are used during the build process, but aren’t preserved at the end of the build; or
(2) Include some concept of the “local compiler” in the build process, which can be used to compile pgen and _freeze_importlib; or
There might be other approaches that will work; as I said, build systems aren’t my strength.
Yours, Russ Magee %-)
Hi Russell. Sorry for the minor ~1 month delay in replying :)
I have been doing some experimenting to see what is involved in cross-compiling Python (Native host = Linux, target = Windows via mingw and some patches). So I have a slightly better understanding of the problem than before.
On 16 February 2016 at 01:41, Russell Keith-Magee russell@keith-magee.com wrote:
In order to build for a host platform, you have to compile for a local platform first - for example, to compile an iOS ARM64 binary, you have to compile for OS X x86_64 first. This gives you a local platform version of Python you can use when building the iOS version.
Early in the Makefile, the variable PYTHON_FOR_BUILD is set. This points at the CPU-local version of Python that can be invoked, which is used for module builds, and for compiling the standard library source code. This is set by —host and —build flags to configure, plus the use of CC and LDFLAGS environment variables to point at the compiler and libraries for the platform you’re compiling for, and a PATH variable that provides the local platform’s version of Python.
So far I haven’t succeeded with my Min GW cross build and am temporarily giving up due to incompatibilities. But my attempts looked a bit like this:
make clean # Work around confusion with existing in-source build mkdir native (cd native/ && ../configure) make -C native/ Parser/pgen mkdir mingw (cd mingw/ && ../configure --host=i486-mingw32 --build=x86) make -C mingw/ PGEN=../native/Parser/pgen
Actually it was not as smooth as the above commands, because pgen tends to get overwritten with a cross-compiled version. Perhaps we could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used at https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html.
There are two places where special handling is required: the compilation and execution of the parser generator, and _freeze_importlib. In both cases, the tool needs to be compiled for the local platform, and then executed. Historically (i.e., Py3.4 and earlier), this has been done by spawning a child MAKE to compile the tool; this runs the compilation phase with the local CPU environment, before returning to the master makefile and executing the tool. By spawning the child MAKE, you get a “clean” environment, so the tool is built natively. However, as I understand it, it causes problems with parallel builds due to race conditions on build rules. The change in Python3.5 simplified the rule so that child MAKE calls weren’t used, but that means that pgen and _freeze_importlib are compiled for ARM64, so they won’t run on the local platform.
You suggest that the child Make command happened to compile pgen etc natively, rather than with the cross compiler. But my understanding is that when you invoke $(MAKE), all the environment variables, configure settings, etc, including the cross compiler, would be inherited by the child.
Would it be more correct to say instead that in 3.4 you did a separate native build step, precompiling pgen and _freeze_importlib for the native build host? Then you hoped that the child Make was _not_ invoked in the cross-compilation stage and your precompiled executables would not be rebuilt?
As best as I can work out, the solution is to:
(1) Include the parser generator and _freeze_importlib as part of the artefacts of local platform. That way, you could use the version of pgen and _freeze_importlib that was compiled as part of the local platform build. At present, pgen and _freeze_importlib are used during the build process, but aren’t preserved at the end of the build; or
I don’t understand. After I run Make, it looks like I get working executables leftover at Programs/_freeze_importlib and Parser/pgen. Do you mean to install these programs with “make install” or something?
(2) Include some concept of the “local compiler” in the build process, which can be used to compile pgen and _freeze_importlib; or
On the surface solution (2) sounds like the ideal fix. But I guess the local native compiler might also require a separate set of CPPFLAGS, pyconfig.h settings etc. In other words it is sounding like a whole separate “configure” run. I am thinking it might be simplest to just require this native “configure” run to be done manually.
On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter vadmium+py@gmail.com wrote:
Hi Russell. Sorry for the minor ~1 month delay in replying :)
I have been doing some experimenting to see what is involved in cross-compiling Python (Native host = Linux, target = Windows via mingw and some patches). So I have a slightly better understanding of the problem than before.
On 16 February 2016 at 01:41, Russell Keith-Magee russell@keith-magee.com wrote:
In order to build for a host platform, you have to compile for a local platform first - for example, to compile an iOS ARM64 binary, you have to compile for OS X x86_64 first. This gives you a local platform version of Python you can use when building the iOS version.
Early in the Makefile, the variable PYTHON_FOR_BUILD is set. This points
at
the CPU-local version of Python that can be invoked, which is used for module builds, and for compiling the standard library source code. This
is
set by —host and —build flags to configure, plus the use of CC and
LDFLAGS
environment variables to point at the compiler and libraries for the platform you’re compiling for, and a PATH variable that provides the
local
platform’s version of Python.
So far I haven’t succeeded with my Min GW cross build and am temporarily giving up due to incompatibilities. But my attempts looked a bit like this:
make clean # Work around confusion with existing in-source build mkdir native (cd native/ && ../configure) make -C native/ Parser/pgen mkdir mingw (cd mingw/ && ../configure --host=i486-mingw32 --build=x86) make -C mingw/ PGEN=../native/Parser/pgen
Actually it was not as smooth as the above commands, because pgen tends to get overwritten with a cross-compiled version. Perhaps we could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used at < https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cr...
.
That might fix the pgen problem, but _freeze_importlib still remains. I
suppose the same thing might be possible for _freeze_importlib as well…
There are two places where special handling is required: the compilation and
execution of the parser generator, and _freeze_importlib. In both cases,
the
tool needs to be compiled for the local platform, and then executed. Historically (i.e., Py3.4 and earlier), this has been done by spawning a child MAKE to compile the tool; this runs the compilation phase with the local CPU environment, before returning to the master makefile and
executing
the tool. By spawning the child MAKE, you get a “clean” environment, so
the
tool is built natively. However, as I understand it, it causes problems
with
parallel builds due to race conditions on build rules. The change in Python3.5 simplified the rule so that child MAKE calls weren’t used, but that means that pgen and _freeze_importlib are compiled for ARM64, so
they
won’t run on the local platform.
You suggest that the child Make command happened to compile pgen etc natively, rather than with the cross compiler. But my understanding is that when you invoke $(MAKE), all the environment variables, configure settings, etc, including the cross compiler, would be inherited by the child.
Would it be more correct to say instead that in 3.4 you did a separate native build step, precompiling pgen and _freeze_importlib for the native build host? Then you hoped that the child Make was _not_ invoked in the cross-compilation stage and your precompiled executables would not be rebuilt?
Yes - as far as I can make out (with my admittedly hazy understanding), that appears to be what is going on. Although it’s not that I “hoped” the build wouldn’t happen on the second pass - it was the behavior that was previously relied, and on was altered.
As best as I can work out, the solution is to:
(1) Include the parser generator and _freeze_importlib as part of the artefacts of local platform. That way, you could use the version of pgen
and
_freeze_importlib that was compiled as part of the local platform build.
At
present, pgen and _freeze_importlib are used during the build process,
but
aren’t preserved at the end of the build; or
I don’t understand. After I run Make, it looks like I get working executables leftover at Programs/_freeze_importlib and Parser/pgen. Do you mean to install these programs with “make install” or something?
Making them part of the installable artefacts would be one option, but they don’t have to be installed, just preserved.
For example, as a nasty hack, I’ve been able to use this approach to get the build working for 3.5. After the native build, I copy _freeze_importlib to a “safe” location. I then copy it back into place prior to the target build. It works, but it’s in no way suitable for a final build.
(2) Include some concept of the “local compiler” in the build process,
which
can be used to compile pgen and _freeze_importlib; or
On the surface solution (2) sounds like the ideal fix. But I guess the local native compiler might also require a separate set of CPPFLAGS, pyconfig.h settings etc. In other words it is sounding like a whole separate “configure” run. I am thinking it might be simplest to just require this native “configure” run to be done manually.
That run is going to happen anyway, since you have to compile and build for the native platform.
Yours, Russ Magee %-)
On 11 March 2016 at 23:16, Russell Keith-Magee russell@keith-magee.com wrote:
On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter vadmium+py@gmail.com wrote:
make clean # Work around confusion with existing in-source build mkdir native (cd native/ && ../configure) make -C native/ Parser/pgen mkdir mingw (cd mingw/ && ../configure --host=i486-mingw32 --build=x86) make -C mingw/ PGEN=../native/Parser/pgen
Actually it was not as smooth as the above commands, because pgen tends to get overwritten with a cross-compiled version. Perhaps we could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used at https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html.
That might fix the pgen problem, but _freeze_importlib still remains. I suppose the same thing might be possible for _freeze_importlib as well…
Yes. I never got up to it failing in my experiments, but I think I would propose a FREEZE_IMPORTLIB override variable for that too.
Would it be more correct to say instead that in 3.4 you did a separate native build step, precompiling pgen and _freeze_importlib for the native build host? Then you hoped that the child Make was _not_ invoked in the cross-compilation stage and your precompiled executables would not be rebuilt?
Yes - as far as I can make out (with my admittedly hazy understanding), that appears to be what is going on. Although it’s not that I “hoped” the build wouldn’t happen on the second pass - it was the behavior that was previously relied, and on was altered.
Do you have a copy/patch/link/etc to the actual commands that you relied on? It’s hard to guess exactly what you were doing that broke without this information.
As best as I can work out, the solution is to:
(1) Include the parser generator and _freeze_importlib as part of the artefacts of local platform. That way, you could use the version of pgen and _freeze_importlib that was compiled as part of the local platform build. At present, pgen and _freeze_importlib are used during the build process, but aren’t preserved at the end of the build; or
I don’t understand. After I run Make, it looks like I get working executables leftover at Programs/_freeze_importlib and Parser/pgen. Do you mean to install these programs with “make install” or something?
Making them part of the installable artefacts would be one option, but they don’t have to be installed, just preserved.
What commands are you running that cause them to not be preserved at the end of the build?
For example, as a nasty hack, I’ve been able to use this approach to get the build working for 3.5. After the native build, I copy _freeze_importlib to a “safe” location. I then copy it back into place prior to the target build. It works, but it’s in no way suitable for a final build.
(2) Include some concept of the “local compiler” in the build process, which can be used to compile pgen and _freeze_importlib; or
On the surface solution (2) sounds like the ideal fix. But I guess the local native compiler might also require a separate set of CPPFLAGS, pyconfig.h settings etc. In other words it is sounding like a whole separate “configure” run. I am thinking it might be simplest to just require this native “configure” run to be done manually.
That run is going to happen anyway, since you have to compile and build for the native platform.
participants (2)
-
Martin Panter -
Russell Keith-Magee