[Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 15 March 2016 at 04:32, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 15 March 2016 at 10:49, Martin Panter <vadmium+py at gmail.com> wrote:
>> One more idea I am considering is to decouple the makefile rules from
>> the main build. So to update the generated files you would have to run
>> a separate command like “make graminit” or “make frozen”. The normal
>> build would never regenerate them; although perhaps it could still
>> result in an error or warning if they appear out of date.
>
> Some of them used to work that way and it's an incredible PITA when
> you actually *are* working on one of the affected bits of the
> interpreter - working on those parts is rare, so if there are special
> cases to remember, you can pretty much guarantee we'll have forgotten
> them by the time we work on that piece again.

Perhaps if we wrapped them all up in a common “make regenerate”
target, so it is only one special case to remember? Maybe you could
include other stuff like “make clinic” in that as well. Or you could
include the special commands in the warning messages.

> However, it would be worth reviewing the explicit dependencies on
> "Makefile" and see whether they could be replaced by dependencies on
> Makefile.pre.in instead. I'm confident that will work just fine for
> the importlib bootstrapping, and I suspect it will work for the other
> pregenerated-and-checked-in files as well.

The problem is not the reference to Makefile. The graminit files do
not depend on Makefile. The bigger problem is that the checked-in
files depend on compiled programs. This is a summary of the current
rules for importlib:

_freeze_importlib.o: _freeze_importlib.c Makefile

_freeze_importlib: _freeze_importlib.o [. . .]
        $(LINKCC) [. . .]

importlib_external.h: _bootstrap_external.py _freeze_importlib
        _freeze_importlib _bootstrap_external.py importlib_external.h

importlib.h: _bootstrap.py _freeze_importlib
        _freeze_importlib _bootstrap.py importlib.h

So importlib.h depends on the _freeze_importlib compiled program (and
only indirectly on Makefile). The makefile says we have to compile
_freeze_importlib before checking if importlib.h is up to date.

Gnu Make has order-only prerequisites
<https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html>,
which it seems we could abuse to do most of what we want. But (1) I’m
not sure if we can restrict ourselves to Gnu Make, and (2) it is a
horrible hack and would always compile _freeze_importlib even if it is
never run.