[Python-Dev] Bug in build system for cross-platform builds

[Russell Keith-Magee]

On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmium+py at gmail.com> wrote:

> Hi Russell. Sorry for the minor ~1 month delay in replying :)
>
> I have been doing some experimenting to see what is involved in
> cross-compiling Python (Native host = Linux, target = Windows via
> mingw and some patches). So I have a slightly better understanding of
> the problem than before.
>
> On 16 February 2016 at 01:41, Russell Keith-Magee
> <russell at keith-magee.com> wrote:
> > In order to build for a host platform, you have to compile for a local
> > platform first - for example, to compile an iOS ARM64 binary, you have to
> > compile for OS X x86_64 first. This gives you a local platform version of
> > Python you can use when building the iOS version.
> >
> > Early in the Makefile, the variable PYTHON_FOR_BUILD is set. This points
> at
> > the CPU-local version of Python that can be invoked, which is used for
> > module builds, and for compiling the standard library source code. This
> is
> > set by —host and —build flags to configure, plus the use of CC and
> LDFLAGS
> > environment variables to point at the compiler and libraries for the
> > platform you’re compiling for, and a PATH variable that provides the
> local
> > platform’s version of Python.
>
> So far I haven’t succeeded with my Min GW cross build and am
> temporarily giving up due to incompatibilities. But my attempts looked
> a bit like this:
>
> make clean  # Work around confusion with existing in-source build
> mkdir native
> (cd native/ && ../configure)
> make -C native/ Parser/pgen
> mkdir mingw
> (cd mingw/ && ../configure --host=i486-mingw32 --build=x86)
> make -C mingw/ PGEN=../native/Parser/pgen
>
> Actually it was not as smooth as the above commands, because pgen
> tends to get overwritten with a cross-compiled version. Perhaps we
> could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used
> at <
> https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html
> >.
>
> That might fix the pgen problem,  but _freeze_importlib still remains. I
suppose the same thing might be possible for _freeze_importlib as well…

> There are two places where special handling is required: the compilation
> and
> > execution of the parser generator, and _freeze_importlib. In both cases,
> the
> > tool needs to be compiled for the local platform, and then executed.
> > Historically (i.e., Py3.4 and earlier), this has been done by spawning a
> > child MAKE to compile the tool; this runs the compilation phase with the
> > local CPU environment, before returning to the master makefile and
> executing
> > the tool. By spawning the child MAKE, you get a “clean” environment, so
> the
> > tool is built natively. However, as I understand it, it causes problems
> with
> > parallel builds due to race conditions on build rules. The change in
> > Python3.5 simplified the rule so that child MAKE calls weren’t used, but
> > that means that pgen and _freeze_importlib are compiled for ARM64, so
> they
> > won’t run on the local platform.
>
> You suggest that the child Make command happened to compile pgen etc
> natively, rather than with the cross compiler. But my understanding is
> that when you invoke $(MAKE), all the environment variables, configure
> settings, etc, including the cross compiler, would be inherited by the
> child.
>
> Would it be more correct to say instead that in 3.4 you did a separate
> native build step, precompiling pgen and _freeze_importlib for the
> native build host? Then you hoped that the child Make was _not_
> invoked in the cross-compilation stage and your precompiled
> executables would not be rebuilt?
>

Yes - as far as I can make out (with my admittedly hazy understanding),
that appears to be what is going on. Although it’s not that I “hoped” the
build wouldn’t happen on the second pass - it was the behavior that was
previously relied, and on was altered.


> > As best as I can work out, the solution is to:
> >
> > (1) Include the parser generator and _freeze_importlib as part of the
> > artefacts of local platform. That way, you could use the version of pgen
> and
> > _freeze_importlib that was compiled as part of the local platform build.
> At
> > present, pgen and _freeze_importlib are used during the build process,
> but
> > aren’t preserved at the end of the build; or
>
> I don’t understand. After I run Make, it looks like I get working
> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
> you mean to install these programs with “make install” or something?
>

Making them part of the installable artefacts would be one option, but they
don’t have to be installed, just preserved.

For example, as a nasty hack, I’ve been able to use this approach to get
the build working for 3.5. After the native build, I copy _freeze_importlib
to a “safe” location. I then copy it back into place prior to the target
build. It works, but it’s in no way suitable for a final build.


> > (2) Include some concept of the “local compiler” in the build process,
> which
> > can be used to compile pgen and _freeze_importlib; or
>
> On the surface solution (2) sounds like the ideal fix. But I guess the
> local native compiler might also require a separate set of CPPFLAGS,
> pyconfig.h settings etc. In other words it is sounding like a whole
> separate “configure” run. I am thinking it might be simplest to just
> require this native “configure” run to be done manually.
>

That run is going to happen anyway, since you have to compile and build for
the native platform.

Yours,
Russ Magee %-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160312/0948ab33/attachment-0001.html>