[Python-Dev] Bug in build system for cross-platform builds

Martin Panter vadmium+py at gmail.com
Fri Mar 11 19:48:07 EST 2016


On 11 March 2016 at 23:16, Russell Keith-Magee <russell at keith-magee.com> wrote:
>
> On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmium+py at gmail.com> wrote:
>> make clean  # Work around confusion with existing in-source build
>> mkdir native
>> (cd native/ && ../configure)
>> make -C native/ Parser/pgen
>> mkdir mingw
>> (cd mingw/ && ../configure --host=i486-mingw32 --build=x86)
>> make -C mingw/ PGEN=../native/Parser/pgen
>>
>> Actually it was not as smooth as the above commands, because pgen
>> tends to get overwritten with a cross-compiled version. Perhaps we
>> could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used
>> at
>> <https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html>.
>>
> That might fix the pgen problem,  but _freeze_importlib still remains. I
> suppose the same thing might be possible for _freeze_importlib as well…

Yes. I never got up to it failing in my experiments, but I think I
would propose a FREEZE_IMPORTLIB override variable for that too.

>> Would it be more correct to say instead that in 3.4 you did a separate
>> native build step, precompiling pgen and _freeze_importlib for the
>> native build host? Then you hoped that the child Make was _not_
>> invoked in the cross-compilation stage and your precompiled
>> executables would not be rebuilt?
>
>
> Yes - as far as I can make out (with my admittedly hazy understanding), that
> appears to be what is going on. Although it’s not that I “hoped” the build
> wouldn’t happen on the second pass - it was the behavior that was previously
> relied, and on was altered.

Do you have a copy/patch/link/etc to the actual commands that you
relied on? It’s hard to guess exactly what you were doing that broke
without this information.

>> > As best as I can work out, the solution is to:
>> >
>> > (1) Include the parser generator and _freeze_importlib as part of the
>> > artefacts of local platform. That way, you could use the version of pgen
>> > and
>> > _freeze_importlib that was compiled as part of the local platform build.
>> > At
>> > present, pgen and _freeze_importlib are used during the build process,
>> > but
>> > aren’t preserved at the end of the build; or
>>
>> I don’t understand. After I run Make, it looks like I get working
>> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> you mean to install these programs with “make install” or something?
>
>
> Making them part of the installable artefacts would be one option, but they
> don’t have to be installed, just preserved.

What commands are you running that cause them to not be preserved at
the end of the build?

> For example, as a nasty hack, I’ve been able to use this approach to get the
> build working for 3.5. After the native build, I copy _freeze_importlib to a
> “safe” location. I then copy it back into place prior to the target build.
> It works, but it’s in no way suitable for a final build.
>
>>
>> > (2) Include some concept of the “local compiler” in the build process,
>> > which
>> > can be used to compile pgen and _freeze_importlib; or
>>
>> On the surface solution (2) sounds like the ideal fix. But I guess the
>> local native compiler might also require a separate set of CPPFLAGS,
>> pyconfig.h settings etc. In other words it is sounding like a whole
>> separate “configure” run. I am thinking it might be simplest to just
>> require this native “configure” run to be done manually.
>
>
> That run is going to happen anyway, since you have to compile and build for
> the native platform.


More information about the Python-Dev mailing list