[Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 13 March 2016 at 01:13, Russell Keith-Magee <russell at keith-magee.com> wrote:
> The patches that I've uploaded to Issue23670 [1] show a full cross-platform
> [1] http://bugs.python.org/issue23670
> build process. After you apply that patch, the iOS directory contains a
> meta-Makefile that manages the build process.

Thanks very much for pointing that out. This has helped me understand
a lot more things. Only now do I realize that the four files generated
by pgen and _freeze_importlib are actually already committed into the
Mercurial repository:

Include/graminit.h
Python/graminit.c
Python/importlib.h
Python/importlib_external.h

A question for other Python developers: Why are these generated files
stored in the repository? The graminit ones seem to have been there
since forever (1990). It seems the importlib ones were there due to a
bootstrapping problem, but now that is solved. Antoine
<https://bugs.python.org/issue14928#msg163048> said he kept them in
the repository on purpose, but I want to know why.

If we ignore the cross compiling use case, would there be any other
consequences of removing these generated files from the repository?
E.g. would it affect the Windows build process?

I have two possible solutions in mind: either remove the generated
files from the repository and always build them, or keep them but do
not automatically regenerate them every build. Since they are
generated files, not source files, I would prefer to remove them, but
I want to know the consequences first.

> On Sat, Mar 12, 2016 at 8:48 AM, Martin Panter <vadmium+py at gmail.com> wrote:
>> On 11 March 2016 at 23:16, Russell Keith-Magee <russell at keith-magee.com>
>> wrote:
>> >
>> > On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmium+py at gmail.com>
>> > wrote:
>> >> I don't understand. After I run Make, it looks like I get working
>> >> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> >> you mean to install these programs with "make install" or something?
>> >
>> >
>> > Making them part of the installable artefacts would be one option, but
>> > they
>> > don't have to be installed, just preserved.
>>
>> What commands are you running that cause them to not be preserved at
>> the end of the build?
>
>
> I don't know - this is where I hit the end of my understanding of the build
> process. All I know for certain is that 3.4.2 doesn't have this problem;
> 3.5.1 does, and Issue22359 (fixed in [3]) is the source of the problem. A
> subsequent fix [4] introduced an additional problem with _freeze_importlib.
>
> [3] https://hg.python.org/cpython/rev/565b96093ec8
> [4] https://hg.python.org/cpython/rev/02e3bf65b2f8

After my realization about the generated files, I think I can solve
this one. Before the changes you identified, the build process
probably thought the generated files were up to date, so it didn't
need to cross-compile pgen or _freeze_importlib. If the generated file
timestamps were out of date (e.g. depending on the order they are
checked out or extracted), the first native build stage would have
fixed them up.