On 14 March 2016 at 13:04, Gregory P. Smith greg@krypto.org wrote:

They should not be regenerated every build, if they are, that seems like a bug in the makefile to me (or else the timestamp checks that make does vs how your code checkout happened). Having them checked in is convenient for cross builds as it is one less thing that needs a build-host-arch build.

It's also two less things to go wrong for folks just wanting to work on the 99.9% of CPython that is neither the Grammar file nor importlib._bootstrap.

I'm trying to remember the problem I was solving in making freezeimportlib.o explicitly depend on the Makefile (while cursing past me for not writing it down in the commit message), and I think the issue was that it wasn't correctly picking up changes to the builtin module list. If that's right, then fixing the dependency to be on "$(srcdir)/Makefile.pre.in" instead of the generated Makefile should keep it from getting confused in the cross-compilation scenario, and also restore the behaviour of using the checked in copy rather than rebuilding it just because you ran "./configure".

Cheers, Nick.

On 03/14/2016 02:26 PM, R. David Murray wrote:

The repo-timestamp problem is addressed by the 'make touch' target.

And yes, checking in these platform-independent artifacts is very intentional: less to build, fewer external dependencies in the build process...you don't need to *have* python to *build* python, which you would have to if they were not checked in.

Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect uses of recursive make. The changeset added executable binaries as prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)). This broke cross-compilation: * the executables do not exist and must be cross-compiled * then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since the prerequisites are newer * but the executables cannot run on the build system

Actually the files need not be re-generated as their timestamps have been setup for that purpose with 'make touch'. So a solution to the problem introduced by this changeset when cross-compiling could be to remove the binaries as prerequisites of these rules and include the recipe of their corresponding rules, the one used to build the executable, into the recipes of the original rule. Also IMHO the Programs/_freeze_importlib.c can be used instead of Programs/_freeze_importlib.o as a prerequisite in the Python/importlib.h rule.

[1] https://hg.python.org/cpython/rev/c2a53aa27cad/ [2] http://bugs.python.org/issue22359

Xavier

On 14 March 2016 at 13:26, R. David Murray rdmurray@bitdance.com wrote:

On Mon, 14 Mar 2016 03:04:08 -0000, "Gregory P. Smith" greg@krypto.org wrote:

...

On Sun, Mar 13, 2016 at 7:41 PM Martin Panter vadmium+py@gmail.com wrote:

...

Include/graminit.h Python/graminit.c Python/importlib.h Python/importlib_external.h

A question for other Python developers: Why are these generated files stored in the repository? [. . .]

They should not be regenerated every build, if they are, that seems like a bug in the makefile to me (or else the timestamp checks that make does vs how your code checkout happened).

The reason the current Python 3 build regenerates some files, is because of the makefile prerequisites. For example, Include/graminit.h currently depends on Parser/pgen, which needs to be compiled for the native build host.

...

Having them checked in is convenient for cross builds as it is one less thing that needs a build-host-arch build.

[. . .] And yes, checking in these platform-independent artifacts is very intentional: less to build, fewer external dependencies in the build process...you don't need to *have* python to *build* python, which you would have to if they were not checked in.

Okay so it sounds like the generated files (more listed in .hgtouch) have to stay. Reasons given:

* Some need Python to generate them (bootstrap problem) * Relied on by Windows build system * General convenience (less build steps, less prerequisites, less things to go wrong)

One more idea I am considering is to decouple the makefile rules from the main build. So to update the generated files you would have to run a separate command like “make graminit” or “make frozen”. The normal build would never regenerate them; although perhaps it could still result in an error or warning if they appear out of date.

On 15 March 2016 at 04:32, Nick Coghlan ncoghlan@gmail.com wrote:

On 15 March 2016 at 10:49, Martin Panter vadmium+py@gmail.com wrote:

...

One more idea I am considering is to decouple the makefile rules from the main build. So to update the generated files you would have to run a separate command like “make graminit” or “make frozen”. The normal build would never regenerate them; although perhaps it could still result in an error or warning if they appear out of date.

Some of them used to work that way and it's an incredible PITA when you actually *are* working on one of the affected bits of the interpreter - working on those parts is rare, so if there are special cases to remember, you can pretty much guarantee we'll have forgotten them by the time we work on that piece again.

Perhaps if we wrapped them all up in a common “make regenerate” target, so it is only one special case to remember? Maybe you could include other stuff like “make clinic” in that as well. Or you could include the special commands in the warning messages.

However, it would be worth reviewing the explicit dependencies on "Makefile" and see whether they could be replaced by dependencies on Makefile.pre.in instead. I'm confident that will work just fine for the importlib bootstrapping, and I suspect it will work for the other pregenerated-and-checked-in files as well.

The problem is not the reference to Makefile. The graminit files do not depend on Makefile. The bigger problem is that the checked-in files depend on compiled programs. This is a summary of the current rules for importlib:

_freeze_importlib.o: _freeze_importlib.c Makefile

_freeze_importlib: _freeze_importlib.o [. . .] $(LINKCC) [. . .]

importlib_external.h: _bootstrap_external.py _freeze_importlib _freeze_importlib _bootstrap_external.py importlib_external.h

importlib.h: _bootstrap.py _freeze_importlib _freeze_importlib _bootstrap.py importlib.h

So importlib.h depends on the _freeze_importlib compiled program (and only indirectly on Makefile). The makefile says we have to compile _freeze_importlib before checking if importlib.h is up to date.

Gnu Make has order-only prerequisites https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html, which it seems we could abuse to do most of what we want. But (1) I’m not sure if we can restrict ourselves to Gnu Make, and (2) it is a horrible hack and would always compile _freeze_importlib even if it is never run.

On 15 March 2016 at 08:04, Nick Coghlan ncoghlan@gmail.com wrote:

On 15 March 2016 at 15:15, Martin Panter vadmium+py@gmail.com wrote:

...

_freeze_importlib.o: _freeze_importlib.c Makefile

_freeze_importlib: _freeze_importlib.o [. . .] $(LINKCC) [. . .]

importlib_external.h: _bootstrap_external.py _freeze_importlib _freeze_importlib _bootstrap_external.py importlib_external.h

importlib.h: _bootstrap.py _freeze_importlib _freeze_importlib _bootstrap.py importlib.h

Ah, I understand now - the fundamental problem is with a checked in file depending on a non-checked-in file, so if you clean out all the native build artifacts when cross-compiling, the makefile will attempt to create target versions of all the helper utilities (pgen, _freeze_importlib, argument clinic, etc).

Would it help to have a "make bootstrap" target that touched all the checked in generated files with build dependencies on non-checked-in files, but only after first touching the expected locations of the built binaries they depend on?

That sounds similar to “make touch”, with a couple differences. One trouble I forsee is the conflict with shared prerequisites. E.g. “make bootstrap” would have to create some dummy object files as prerequisites of the pgen program, but we would first have build others e.g. Parser/acceler.o properly for the main Python library. It all feels way too complicated to me. The Python build system is complicated enough as it is.

Maybe it is simplest to just add something in the spirit of Xavier’s suggested patch. This would mean that we keep the generated files checked in (to help with Windows and cross compiled builds), we keep the current rules that force normal makefile builds to blindly regenerate the files, but we add some flag or configure.ac check to disable this regeneration if desired.